[Tails-dev] Tails control port filter proxy in Whonix?

[Patrick Schleizer]

Hi,

as discussed elsewhere, yes, it would be great if we could share code bases!

Does it support simultaneous connections? (Such as two applications
using ephemeral Tor hidden services plus Tor Browser at once.)

Does Tails control port filter proxy support events? I mean, can a
client application ask for something and Tor will maybe answer a long
time later?

Whonix control-port-filter-python TODO, also stuff we need before we can

- https://phabricator.whonix.org/T561

Is something we must use in Whonix. Not a cpfpy missing feature but a
general issue. In essence, for example the onionshare localhost server
listener will not be reachable. We somehow must force it listen on all
interfaces so Tor running on the gateway can access it.

- https://phabricator.whonix.org/T562

This is about parsing add_onion and whitelisting sane commands rather
than letting through everything.

add_onion is not a whitelist/not whitelist.

Buggy applications or by user mistake, they could choose the add_onion
flag nonanonymous, which would be a disaster. We also don't know what
Tor control protocol upgrades are coming in the years to come. So I
strongly suggest a only letting through whitelisted syntaxes.

Malicious applications could make the Tor HS listener bind on the wrong
interface. In Whonix-Gateway, maliciously listen on Whonix-Gateway.
Which could be fatal if we had also a real Tor ControlPort open there.
Does that make sense? I am not sure it applies to Tails, that depends on
your design and threat model, but it is however an interesting thought
that can inspire to finding more security issues with it.

Also it may be worth making sure it can only bind to specified (and
configureable) local ports?

For connectivity, we need to remove 127.0.0.1 and replace it with
Whonix-Workstation IP. That is currently done with the following code
block that I was going to merge with T562.

https://github.com/Whonix/control-port-filter-python/blob/6a131266a8dc8f98ff22a3b83fae9d43e38b3127/usr/sbin/cpfpd#L345-L375

- https://phabricator.whonix.org/T564

Protecting cpfpy from DDOS from client applications. Not sure that
matters for Tails?

- https://phabricator.whonix.org/T565

Similar to above.

- https://phabricator.whonix.org/T566

The unit test for T562.

Other required features:

- Configurable by dropping .d-style[7] configuration snippets. (ex:
/etc/cpfpy.d)
- Debian packaging.

Lesser important features:

- Supports logging.
- Honors signals sigterm, sigint, keyboard interrupt.
- systemd support
- When request is 'getinfo net/listeners/socks' answer with a lie
'250-net/listeners/socks="127.0.0.1:9150"'.

Cheers,
Patrick

[anonym]

Post by Patrick Schleizer
Hi,
as discussed elsewhere, yes, it would be great if we could share code bases!

Agreed, but we have to realize that at the moment Whonix and Tails run
these filters in quite different contexts and under different threat
models. Whonix runs the filter in a different VM than the clients
connecting to it, and so has less information about the client
connections than what we can have in Tails (e.g. client PID/uid/gid,
executable path, blah) and probably most problems you are facing are due
to this compartmentalization. Really, what makes Tails' control port
filter worthwhile is it's ability to match filter rules to the client
executable/script, and since Whonix won't be able to make use of this (I
mean, the appvm is untrusted in your threat model, so any such
information cannot be trusted any way) there's perhaps not so much to
gain to migrate. :/

I recommend that you read our monthly report for September:

https://tails.boum.org/news/report_2016_09/#index2h1

and look at the documentation at the top of the script, and the filter
rules we ship to get an idea of what it can do.

As you can see, in Tails we use match-exe-paths and match-users a lot,
but since you won't have access to these I guess you want something like
match-hosts, so that the filter is picked based on the client's
(non-localhost) address (= VM). Right?

Post by Patrick Schleizer
Does it support simultaneous connections? (Such as two applications
using ephemeral Tor hidden services plus Tor Browser at once.)

Yes, it uses socketserver.ThreadingTCPServer with each such server
thread opening its own "real" control port session with tor.

Post by Patrick Schleizer
Does Tails control port filter proxy support events? I mean, can a
client application ask for something and Tor will maybe answer a long
time later?

Yes, thanks to python-stem it was simple.

Post by Patrick Schleizer
Whonix control-port-filter-python TODO, also stuff we need before we can

- https://phabricator.whonix.org/T561

Is something we must use in Whonix. Not a cpfpy missing feature but a
general issue. In essence, for example the onionshare localhost server
listener will not be reachable. We somehow must force it listen on all
interfaces so Tor running on the gateway can access it.

Got it. This seems like a problem orthogonal to the filter, so I skip it.

Post by Patrick Schleizer

- https://phabricator.whonix.org/T562

This is about parsing add_onion and whitelisting sane commands rather
than letting through everything.

For any command we allow a list of regexes for the arguments. If a
command doesn't match any of them, it is filtered.

Post by Patrick Schleizer
add_onion is not a whitelist/not whitelist.

I do not understand this sentence...

Post by Patrick Schleizer
Buggy applications or by user mistake, they could choose the add_onion
flag nonanonymous, which would be a disaster. We also don't know what
Tor control protocol upgrades are coming in the years to come. So I
strongly suggest a only letting through whitelisted syntaxes.

... but this I get and agree with. Currently we require ADD_ONION for
onionshare to have args matching the regex 'NEW:BEST Port=80,176\d\d'.

Post by Patrick Schleizer
Malicious applications could make the Tor HS listener bind on the wrong
interface. In Whonix-Gateway, maliciously listen on Whonix-Gateway.
Which could be fatal if we had also a real Tor ControlPort open there.
Does that make sense? I am not sure it applies to Tails, that depends on
your design and threat model, but it is however an interesting thought
that can inspire to finding more security issues with it.
Also it may be worth making sure it can only bind to specified (and
configureable) local ports?

Ack, this is an issue. In Tails there must be an explicit firewall rule
allowing a user listening on some port, so I think we are covered.

Post by Patrick Schleizer
For connectivity, we need to remove 127.0.0.1 and replace it with
Whonix-Workstation IP. That is currently done with the following code
block that I was going to merge with T562.
https://github.com/Whonix/control-port-filter-python/blob/6a131266a8dc8f98ff22a3b83fae9d43e38b3127/usr/sbin/cpfpd#L345-L375

Got it. Our filter doesn't do this (as we do not have this need) but I
feel a general solution could be to allow sed-like rewriting rules, e.g.

ADD_ONION:
- pattern: 'NEW:BEST Port=80,(176\d\d)'
replacement: 'NEW:BEST Port=80,10.137.6.41:{0}'

which would be easy to implement.

Post by Patrick Schleizer

- https://phabricator.whonix.org/T564

Protecting cpfpy from DDOS from client applications. Not sure that
matters for Tails?

We do not do much specific here. What kind of DoS are you talking about
here? Eating up all RAM or crashing the filter via oom kill? Preventing
the filter from serving other clients? We admittedly do not do much here
except that each client is dealt with in a separate thread, and that
client requests are limited to 1024 bytes.

Post by Patrick Schleizer

- https://phabricator.whonix.org/T565

Similar to above.

There's quite some cost in dealing with each new client since we do a
client port -> client PID lookup, but this wouldn't be an issue in
Whonix. Still, flooding the filter will probably lead to huge CPU usage
no matter what. Tor itself is susceptible to this, but I'm sure the
control port filter adds at least 10x overhead.

Post by Patrick Schleizer

- https://phabricator.whonix.org/T566

The unit test for T562.

Writing unit tests for commands vs rules should be easy, around
handle_controlport_session() -- just pass it the rules, a read socket
that you write the commands you want to test to, and a write socket that
you read back the result from.

Post by Patrick Schleizer
/etc/cpfpy.d)

Yes.

Post by Patrick Schleizer
- Debian packaging.

No, but if you plan to use it, I guess I'd give a shot at packaging it.

Post by Patrick Schleizer
- Supports logging.

I'm not sure what type of logging you are talking about here. Currently
it uses print() (to stdout) and flush, which works well with the journal
when run as a systemd service. Normally only filtered commands are
logged, but there's also a --complain mode which logs all client
requests, which is useful when writing rules for a new application.

Post by Patrick Schleizer
- Honors signals sigterm, sigint, keyboard interrupt.

I guess?

Post by Patrick Schleizer
- systemd support

Not sure what this means. We have a .service file.

Post by Patrick Schleizer
- When request is 'getinfo net/listeners/socks' answer with a lie
'250-net/listeners/socks="127.0.0.1:9150"'.

Nope. Why is this needed?

I could even imagine yet another rule-type for solving these types of
issues:

GETINFO:
- pattern: 'net/listeners/socks'
respond: '250-net/listeners/socks="127.0.0.1:9150"'

--

In conclusion, I think the truth is that Whonix switching to our filter
will require some work to reach feature-parity with you current filter,
and you will not really gain anything by doing so except code sharing.
YMMV. That said, I'd happily implement match-hosts and the two
additional types of rules I mentioned above if that would be enough for you.

Still, I feel we've left out roflcoptor from this discussion. At the
moment the biggest turn-off for me is that it is written in Go, a
language I have little interest in learning, and the lack of Debian
packaging. I'm not sure what else to say, but it just feels like there
needs to be a discussion before we'd proceed in collaborating on a
control port filter.

Cheers!

[anonym]

Post by anonym

Post by Patrick Schleizer
Hi,
as discussed elsewhere, yes, it would be great if we could share code bases!

I got excited about doing some coding, so now we're closer to being able
to do this! Still, don't feel bad if you in the end decide *not* to use
our filter. :)

[...]

Post by anonym

Post by Patrick Schleizer
- Honors signals sigterm, sigint, keyboard interrupt.

I guess?

Now it definitely does.

[...]

Post by anonym
In conclusion, I think the truth is that Whonix switching to our filter
will require some work to reach feature-parity with you current filter,
and you will not really gain anything by doing so except code sharing.
YMMV. That said, I'd happily implement match-hosts and the two
additional types of rules I mentioned above if that would be enough for you.

I actually went ahead and implemented these, and I've successfully been
able to run a client on a different host than the filter, like in
Whonix. This is how I imagine the onionshare filter configuration Whonix
needs would look like:

- match-hosts:
- '10.1.1.42'
commands:
GETINFO:
- 'version'
- 'onions/current'
- pattern: 'net/listeners/socks'
response: '250-net/listeners/socks="127.0.0.1:9150"'
GETCONF:
- '__owningcontrollerprocess'
ADD_ONION:
- pattern: 'NEW:BEST Port=80,(176\d\d)'
replacement: 'NEW:BEST Port=80,10.137.6.41:{}'
DEL_ONION:
- '.+'
events:
SIGNAL:
suppress: true
CONF_CHANGED:
suppress: true
HS_DESC:

If you need help deciphering the above I refer you to the "docs":


http://git.tails.boum.org/tails/tree/config/chroot_local-includes/usr/local/lib/tor-controlport-filter?h=feature/7870-include_onionshare

Thoughts about the configuration format are highly welcome (but I doubt
I'd like to move away from YAML)!

It would be interesting to see if this works for you in Whonix. You
definitely will have to look at the --listen-address option (0.0.0.0 or
the address the gateway connects to), and maybe the
--control-cookie-path and --control-socket-path options (yes, the filter
requires a cookie protected Tor ControlSoket for now -- patches welcome
for other modes!).

Post by anonym
Still, I feel we've left out roflcoptor from this discussion. At the
moment the biggest turn-off for me is that it is written in Go, a
language I have little interest in learning, and the lack of Debian
packaging. I'm not sure what else to say, but it just feels like there
needs to be a discussion before we'd proceed in collaborating on a
control port filter.

I've now seen that you raised similar concerns a bit earlier in the
thread, so perhaps that is good enough for dismissing roflcoptor (at
least for now) without feeling too bad due to NIH syndrome.

I don't know if/how we should proceed, but here's what I think of the
remaining (according to an earlier email in the thread) tickets you have

Post by anonym
- https://phabricator.whonix.org/T561

Orthogonal to the filter, so skipped.

Post by anonym
- https://phabricator.whonix.org/T562

Solved!

Post by anonym
- https://phabricator.whonix.org/T564

I'd need more details of what the idea is here.

Any way, it seems that if one is careful when writing the rules for
ADD_ONION, one can limit the number simply by making sure there's that
number of possible matches of [host:]ports. E.g. the above filter for
onionshare can at most allow 100 ports. Solved?

Post by anonym
- https://phabricator.whonix.org/T565

I'd need more details of what the idea is here.

Post by anonym
- https://phabricator.whonix.org/T566

Like I said, this should be easy. Patches welcome! :)

---

Let me end with: if you test our filter in Whonix, and decide it's the
way to go, then I'll try to do the Debian packaging unless someone wants
to contribute it!

Cheers!

[anonym]

Post by anonym
This is how I imagine the onionshare filter configuration Whonix
- '10.1.1.42'
- 'version'
- 'onions/current'
- pattern: 'net/listeners/socks'
response: '250-net/listeners/socks="127.0.0.1:9150"'
- '__owningcontrollerprocess'
- pattern: 'NEW:BEST Port=80,(176\d\d)'
replacement: 'NEW:BEST Port=80,10.137.6.41:{}'
- '.+'
suppress: true
suppress: true

I've pushed some improvements, in particular for making the rewriting
system more general; now we can rewrite commands before they are sent,
rewrite responses from commands (even a rewritten one!), and rewrite
received events. Because of this the profile above needs a slight
adjustment (the only change is for `GETINFO net/listeners/socks`):

- match-hosts:
- '10.1.1.42'
commands:
GETINFO:
- 'version'
- 'onions/current'
- pattern: 'net/listeners/socks'
response:
- pattern: '.*'
replacement: '250-net/listeners/socks="127.0.0.1:9150"'
ADD_ONION:
- pattern: 'NEW:BEST Port=80,(176([0-4][0-]|50))'
replacement: 'NEW:BEST Port=80,10.137.6.41:{}'
DEL_ONION:
- '.+'
confs:
__owningcontrollerprocess:
events:
SIGNAL:
suppress: true
CONF_CHANGED:
suppress: true
HS_DESC:

... in case you want to test it in Whonix. :)

To highlight the possibilities of the rewriting system, we now allow
rules like this to fool an application about what hosts are connected to
through tor:

commands:
GETINFO:
- pattern: 'stream-status'
response:
- pattern: '(.*)whonix\.org(.*)'
replacement: '{}tails.boum.org{}'
- pattern: '(.*)tails\.boum\.org(.*)'
replacement: '{}whonix.org{}'
events:
STREAM:
response:
[ Same two (pattern, replacement)-pairs as above ]

Only one replacement will ever be done on the same string, and the
matching is done from top to bottom, as defined in the configuration
file. With the empty string as `replacement`, the matched response would
simply be dropped instead. If we replace the above (pattern,
replacement)-pairs with only:

pattern: '(\d+ STREAM \d+ \S+ \d+) [^:]+:(.+)'
replacement: '{} tails.boum.org:{}'

we fool the application where all tor streams go. The question is,
however, whether this is a useful feature at all. :)

Cheers!

[intrigeri]

Hi,

maybe this discussion would be one of those that we should be having
on the ***@secure-os.org list?

I see that there was a thread about similar topics a few months ago:
https://secure-os.org/pipermail/desktops/2016-July/000128.html

Cheers,

[Patrick Schleizer]

Post by anonym
[...]

Post by anonym
In conclusion, I think the truth is that Whonix switching to our filter
will require some work to reach feature-parity with you current filter,
and you will not really gain anything by doing so except code sharing.
YMMV. That said, I'd happily implement match-hosts and the two
additional types of rules I mentioned above if that would be enough for you.

I actually went ahead and implemented these, and I've successfully been
able to run a client on a different host than the filter, like in
Whonix. This is how I imagine the onionshare filter configuration Whonix
- '10.1.1.42'
- 'version'
- 'onions/current'
- pattern: 'net/listeners/socks'
response: '250-net/listeners/socks="127.0.0.1:9150"'
- '__owningcontrollerprocess'
- pattern: 'NEW:BEST Port=80,(176\d\d)'
replacement: 'NEW:BEST Port=80,10.137.6.41:{}'
- '.+'
suppress: true
suppress: true
http://git.tails.boum.org/tails/tree/config/chroot_local-includes/usr/local/lib/tor-controlport-filter?h=feature/7870-include_onionshare
Thoughts about the configuration format are highly welcome (but I doubt
I'd like to move away from YAML)!

Looks good!

Post by anonym
It would be interesting to see if this works for you in Whonix. You
definitely will have to look at the --listen-address option (0.0.0.0 or
the address the gateway connects to), and maybe the
--control-cookie-path and --control-socket-path options (yes, the filter
requires a cookie protected Tor ControlSoket for now -- patches welcome
for other modes!).

Yes, no longer using ControlPort is a very good path. ControlSocket just
matches the defaults also in Whonix.

Post by anonym
I don't know if/how we should proceed, but here's what I think of the
remaining (according to an earlier email in the thread) tickets you have

Post by anonym
- https://phabricator.whonix.org/T562

Solved!

Post by anonym
- https://phabricator.whonix.org/T564

I'd need more details of what the idea is here.

Prevent (in case of some bug or compromise) that more than X hidden
services are created. The number of hidden service should be tracked. If
more than X are created, requests for creating even more should be rejected.

This is because too many hidden services on the same connection may be
suspicious for ISP level adversaries or even dangerous for other tricky
anonymity reasons.

Post by anonym
Any way, it seems that if one is careful when writing the rules for
ADD_ONION, one can limit the number simply by making sure there's that
number of possible matches of [host:]ports. E.g. the above filter for
onionshare can at most allow 100 ports. Solved?

I guess. Is that already possible?

Post by anonym

Post by anonym
- https://phabricator.whonix.org/T565

I'd need more details of what the idea is here.

Have a script (or even unit test) that runs a loop, that creates a
zillion of hidden services. And then check if this has the chance to
make the whole system unusably slow. That would be considered ddos, and bad.

Post by anonym
---
Let me end with: if you test our filter in Whonix, and decide it's the
way to go, then I'll try to do the Debian packaging unless someone wants
to contribute it!

That's an awesome offer you! Yes, please do.

Cheers,
Patrick

[anonym]

Post by Patrick Schleizer

Post by anonym

- https://phabricator.whonix.org/T564

I'd need more details of what the idea is here.

Prevent (in case of some bug or compromise) that more than X hidden
services are created. The number of hidden service should be tracked. If
more than X are created, requests for creating even more should be rejected.
This is because too many hidden services on the same connection may be
suspicious for ISP level adversaries or even dangerous for other tricky
anonymity reasons.

Ok, while the "other tricky anonymity reasons" may be true, I don't buy
the ISP argument: in its position (it sees the first hop) the traffic
generated by hosting lots of (never unused) hidden services won't look
very interesting, I'd wager. If it looked interesting, an attacker that
has compromised Whonix workstation would most likely be able to generate
the same traffic pattern by using tor in some other way. Any way, I
think a proper threat model where e.g. "suspicious traffic" is defined
is needed to treat this properly. IMHO we have tons of much lower
hanging fruit in the area of "network fingerprinting".

Post by Patrick Schleizer

Post by anonym
Any way, it seems that if one is careful when writing the rules for
ADD_ONION, one can limit the number simply by making sure there's that
number of possible matches of [host:]ports. E.g. the above filter for
onionshare can at most allow 100 ports. Solved?

I guess. Is that already possible?

Yes, e.g.:

ADD_ONION:
- pattern: 'NEW:BEST Port=80,(176[0-5][0-9])'

That would match only the 50 different ports that onionshare will ever
try to run the web server on. It's a solution by coincidence, but it
works! :)

Post by Patrick Schleizer

Post by anonym

- https://phabricator.whonix.org/T565

I'd need more details of what the idea is here.

Have a script (or even unit test) that runs a loop, that creates a
zillion of hidden services. And then check if this has the chance to
make the whole system unusably slow. That would be considered ddos, and bad.

onionshare (with and without the filter) is limited to 50 of them at the
same time, so this does not seem like a concern for now. Just for kicks
and giggles I tried running

for i in $(seq 1 100); do echo $i > $i; onionshare $i & done

but this just exposed that onionshare's method of picking a free local
port is extremely prone to a race condition (many instances picking the
same port) so the results are not very relevant.

Post by Patrick Schleizer

Post by anonym
Let me end with: if you test our filter in Whonix, and decide it's the
way to go, then I'll try to do the Debian packaging unless someone wants
to contribute it!

That's an awesome offer you! Yes, please do.

Sorry, what I meant was: if you *first* try it (which should be easy in
a running Whonix session) and decide that you want to use it, *then* I
will do the packaging. Fair enough?

Cheers!

[Patrick Schleizer]

Post by anonym

Post by Patrick Schleizer

Post by anonym

- https://phabricator.whonix.org/T564

I'd need more details of what the idea is here.

Prevent (in case of some bug or compromise) that more than X hidden
services are created. The number of hidden service should be tracked. If
more than X are created, requests for creating even more should be rejected.
This is because too many hidden services on the same connection may be
suspicious for ISP level adversaries or even dangerous for other tricky
anonymity reasons.

Ok, while the "other tricky anonymity reasons" may be true, I don't buy
the ISP argument: in its position (it sees the first hop) the traffic
generated by hosting lots of (never unused) hidden services won't look
very interesting, I'd wager. If it looked interesting, an attacker that
has compromised Whonix workstation would most likely be able to generate
the same traffic pattern by using tor in some other way. Any way, I
think a proper threat model where e.g. "suspicious traffic" is defined
is needed to treat this properly. IMHO we have tons of much lower
hanging fruit in the area of "network fingerprinting".

I agree with this.

Post by anonym

Post by Patrick Schleizer

Post by anonym
Any way, it seems that if one is careful when writing the rules for
ADD_ONION, one can limit the number simply by making sure there's that
number of possible matches of [host:]ports. E.g. the above filter for
onionshare can at most allow 100 ports. Solved?

I guess. Is that already possible?

- pattern: 'NEW:BEST Port=80,(176[0-5][0-9])'
That would match only the 50 different ports that onionshare will ever
try to run the web server on. It's a solution by coincidence, but it
works! :)

Ha, tricky, but a good solution.

Post by anonym

Post by Patrick Schleizer

Post by anonym
Let me end with: if you test our filter in Whonix, and decide it's the
way to go, then I'll try to do the Debian packaging unless someone wants
to contribute it!

That's an awesome offer you! Yes, please do.

Sorry, what I meant was: if you *first* try it (which should be easy in
a running Whonix session) and decide that you want to use it, *then* I
will do the packaging. Fair enough?

Sure. I was already testing the filter inside a running Whonix session. :)

So for using the Tails control port filter proxy in Whonix, there are
just two remaining blockers. The Debian packaging and the following...

Post by anonym

Post by Patrick Schleizer
For connectivity, we need to remove 127.0.0.1 and replace it with

Post by anonym

Whonix-Workstation IP. That is currently done with the following code
block that I was going to merge with T562.

https://github.com/Whonix/control-port-filter-python/blob/6a131266a8dc8f98ff22a3b83fae9d43e38b3127/usr/sbin/cpfpd#L345-L375

Post by anonym

Post by Patrick Schleizer
Got it. Our filter doesn't do this (as we do not have this need) but I
feel a general solution could be to allow sed-like rewriting rules, e.g.
- pattern: 'NEW:BEST Port=80,(176\d\d)'
replacement: 'NEW:BEST Port=80,10.137.6.41:{0}'
which would be easy to implement.

Where I need to correct myself. The injected IP is probably difficult to
add to a config file since IPs in Qubes will remain dynamic for some
quite some time until Qubes 4.0. We'd need something like this.

ADD_ONION:
- pattern: 'NEW:BEST Port=80,(176[0-5][0-9])'
replacement: 'NEW:BEST Port=80,<client i.e. workstation IP>:{}'

(Where <workstation IP> is just used to illustrate. Not a syntax
suggestion. Could be expressed with any other special chars.)

Could you implement that please?

Cheers,
Patrick

[anonym]

Post by Patrick Schleizer
Where I need to correct myself. The injected IP is probably difficult to
add to a config file since IPs in Qubes will remain dynamic for some
quite some time until Qubes 4.0. We'd need something like this.
- pattern: 'NEW:BEST Port=80,(176[0-5][0-9])'
replacement: 'NEW:BEST Port=80,<client i.e. workstation IP>:{}'
(Where <workstation IP> is just used to illustrate. Not a syntax
suggestion. Could be expressed with any other special chars.)
Could you implement that please?

I hacked something together so that the following should work for you:

ADD_ONION:
- pattern: 'NEW:BEST Port=80,(176[0-5][0-9])'
replacement: 'NEW:BEST Port=80,{client-address}:{}'

See attached patch, but note that I haven't tested it (and not pushed
it, since the branch is up for review, and I won't have time to test it
for that). If there's some silly syntax error, I bet you can fix it
yourself. :)

Cheers!

[Patrick Schleizer]

Post by anonym

Post by Patrick Schleizer
Where I need to correct myself. The injected IP is probably difficult to
add to a config file since IPs in Qubes will remain dynamic for some
quite some time until Qubes 4.0. We'd need something like this.
- pattern: 'NEW:BEST Port=80,(176[0-5][0-9])'
replacement: 'NEW:BEST Port=80,<client i.e. workstation IP>:{}'
(Where <workstation IP> is just used to illustrate. Not a syntax
suggestion. Could be expressed with any other special chars.)
Could you implement that please?

- pattern: 'NEW:BEST Port=80,(176[0-5][0-9])'
replacement: 'NEW:BEST Port=80,{client-address}:{}'
See attached patch, but note that I haven't tested it (and not pushed
it, since the branch is up for review, and I won't have time to test it
for that). If there's some silly syntax error, I bet you can fix it
yourself. :)

Fixed some minor issues indeed. Patch attached.

However, there is an offending line, I am stuck with.

return r['replacement'].format(*match.groups()) + terminator

File "./tor-controlport-filter", line 334, in rewrite_line
return r['replacement'].format(*match.groups()) + terminator
KeyError: 'client-address'

Could you fix that please?

Cheers,
Patrick

[anonym]

Post by Patrick Schleizer

Post by anonym

Post by Patrick Schleizer
Where I need to correct myself. The injected IP is probably difficult to
add to a config file since IPs in Qubes will remain dynamic for some
quite some time until Qubes 4.0. We'd need something like this.
- pattern: 'NEW:BEST Port=80,(176[0-5][0-9])'
replacement: 'NEW:BEST Port=80,<client i.e. workstation IP>:{}'
(Where <workstation IP> is just used to illustrate. Not a syntax
suggestion. Could be expressed with any other special chars.)
Could you implement that please?

- pattern: 'NEW:BEST Port=80,(176[0-5][0-9])'
replacement: 'NEW:BEST Port=80,{client-address}:{}'
See attached patch, but note that I haven't tested it (and not pushed
it, since the branch is up for review, and I won't have time to test it
for that). If there's some silly syntax error, I bet you can fix it
yourself. :)

Fixed some minor issues indeed. Patch attached.
However, there is an offending line, I am stuck with.
return r['replacement'].format(*match.groups()) + terminator
File "./tor-controlport-filter", line 334, in rewrite_line
return r['replacement'].format(*match.groups()) + terminator
KeyError: 'client-address'
Could you fix that please?

Yesterday's patch was trash. See the new commit(s) I've just pushed to
the branch.

Cheers!

[Patrick Schleizer]

Post by anonym

Post by Patrick Schleizer

Post by anonym

Post by Patrick Schleizer
Where I need to correct myself. The injected IP is probably difficult to
add to a config file since IPs in Qubes will remain dynamic for some
quite some time until Qubes 4.0. We'd need something like this.
- pattern: 'NEW:BEST Port=80,(176[0-5][0-9])'
replacement: 'NEW:BEST Port=80,<client i.e. workstation IP>:{}'
(Where <workstation IP> is just used to illustrate. Not a syntax
suggestion. Could be expressed with any other special chars.)
Could you implement that please?

- pattern: 'NEW:BEST Port=80,(176[0-5][0-9])'
replacement: 'NEW:BEST Port=80,{client-address}:{}'
See attached patch, but note that I haven't tested it (and not pushed
it, since the branch is up for review, and I won't have time to test it
for that). If there's some silly syntax error, I bet you can fix it
yourself. :)

Fixed some minor issues indeed. Patch attached.
However, there is an offending line, I am stuck with.
return r['replacement'].format(*match.groups()) + terminator
File "./tor-controlport-filter", line 334, in rewrite_line
return r['replacement'].format(*match.groups()) + terminator
KeyError: 'client-address'
Could you fix that please?

Yesterday's patch was trash. See the new commit(s) I've just pushed to
the branch.

That seems to work! :)

When the filter is terminated, onionshare apparently does not notice
that. Would be better if onionshare would notice that. Is that a bug?

About the packaging. If you like the genmkfile way to package things, I
could also do the packaging. Only disadvantage would be an extra
dependency on genmkfile.

https://github.com/Whonix/control-port-filter-python
https://github.com/Whonix/genmkfile

That would be trivial on my side since Tails control filter seems very
similar to control-port-filter-python. (control-port-filter-python
packaged with genmkfile is a lintian --pedantic warning free and to my
knowledge, fully complaint Debian source and binary package.) Otherwise,
I'd just wait for you.

I added 'Flags=DiscardPK', which works and I thought that is useful at
least in case of Whonix. The workstation does not need to learn the
hidden service key since onionshare does not use it anyhow. Not sure
this (and the following) makes also sense in Tails?

Some lines in Tor's response contain the following:
(azazazazazazaz10 is the HS)

650 HS_DESC CREATED azazazazazazaz10 UNKNOWN UNKNOWN gliberrish REPLICA=0
650 HS_DESC CREATED azazazazazazaz10 UNKNOWN UNKNOWN gliberrish REPLICA=1
650 HS_DESC UPLOAD azazazazazazaz10 UNKNOWN gliberrish gliberrish
650 HS_DESC UPLOADED azazazazazazaz10 UNKNOWN gliberrish

Could you please show how to rewrite them to:

650 HS_DESC CREATED azazazazazazaz10 UNKNOWN UNKNOWN dedacted REPLICA=0
650 HS_DESC CREATED azazazazazazaz10 UNKNOWN UNKNOWN dedacted REPLICA=1
650 HS_DESC UPLOAD azazazazazazaz10 UNKNOWN dedacted dedacted
650 HS_DESC UPLOADED azazazazazazaz10 UNKNOWN dedacted

I am not sure stem would complain about this, but I guess not and seems
useful to me be to contain that information.

Cheers,
Patrick

[anonym]

Post by Patrick Schleizer

Post by anonym
[...]
Yesterday's patch was trash. See the new commit(s) I've just pushed to
the branch.

That seems to work! :)

Woo!

Post by Patrick Schleizer
When the filter is terminated, onionshare apparently does not notice
that. Would be better if onionshare would notice that. Is that a bug?

It seems like a "bug" in onionshare, or even the control port language
protocol itself since it (AFAICT) doesn't have a concept of the server
quitting mid-session. No signal is sent, and I haven't even found an
event one can explicitly subscribe to to learn when it shuts down. In
fact, any stem-application will, for instance, notice that Tor closed
its control port on the next send() or recv() on the socket, and then
throw a stem.SocketClosed.

Post by Patrick Schleizer
About the packaging. If you like the genmkfile way to package things, I
could also do the packaging. Only disadvantage would be an extra
dependency on genmkfile.
https://github.com/Whonix/control-port-filter-python
https://github.com/Whonix/genmkfile
That would be trivial on my side since Tails control filter seems very
similar to control-port-filter-python. (control-port-filter-python
packaged with genmkfile is a lintian --pedantic warning free and to my
knowledge, fully complaint Debian source and binary package.) Otherwise,
I'd just wait for you.

Sure! I bet you'll beat me to it, so I don't want to be the blocker, but
perhaps I'll take over the packaging in the future, depending on how
things go (would this make sense to include in Debian?). However, first
we need, in order:

1. A name that doesn't including 'tor'. What about 'grater' or
'onion-grater'? My inspiration is something like [0]. :) Otherwise
there's 'onion-filter' I guess.

2. A dedicated Git repo (trivial once we have a name).

[0] Loading Image...

Post by Patrick Schleizer
I added 'Flags=DiscardPK', which works and I thought that is useful at
least in case of Whonix. The workstation does not need to learn the
hidden service key since onionshare does not use it anyhow. Not sure
this (and the following) makes also sense in Tails?
(azazazazazazaz10 is the HS)
650 HS_DESC CREATED azazazazazazaz10 UNKNOWN UNKNOWN gliberrish REPLICA=0
650 HS_DESC CREATED azazazazazazaz10 UNKNOWN UNKNOWN gliberrish REPLICA=1
650 HS_DESC UPLOAD azazazazazazaz10 UNKNOWN gliberrish gliberrish
650 HS_DESC UPLOADED azazazazazazaz10 UNKNOWN gliberrish
650 HS_DESC CREATED azazazazazazaz10 UNKNOWN UNKNOWN dedacted REPLICA=0
650 HS_DESC CREATED azazazazazazaz10 UNKNOWN UNKNOWN dedacted REPLICA=1
650 HS_DESC UPLOAD azazazazazazaz10 UNKNOWN dedacted dedacted
650 HS_DESC UPLOADED azazazazazazaz10 UNKNOWN dedacted
I am not sure stem would complain about this, but I guess not

onionshare uses stem's `create_ephemeral_hidden_service()` convenience
method with `await_publication=True`, and it tracks the state of the
hidden services publication via HS_DESC events, so you have to be
careful here (e.g. only allowing the UPLOADED won't work, probably).

So, to me it looks like you need the following (untested) config,
although you might want to study the spec to see which events are to be
expected:

HS_DESC:
- pattern: '650 HS_DESC CREATED (\S+) (\S+) (\S+) \S+ (.+)'
replacement: '650 HS_DESC CREATED {} {} {} redacted {}'
- pattern: '650 HS_DESC UPLOAD (\S+) (\S+) .*'
replacement: '650 HS_DESC UPLOAD {} {} redacted redacted'
- pattern: '650 HS_DESC UPLOADED (\S+) (\S+) .+'
replacement: '650 HS_DESC UPLOADED {} {} redacted'
- pattern: '.*'
replacement: ''

However, I saw something in the stem code that might indicate it
actually uses the value you want to redact, but I might be mistaken.

Post by Patrick Schleizer
and seems useful to me be to contain that information.

Sure!

Cheers!

[Patrick Schleizer]

Post by anonym

Post by Patrick Schleizer
When the filter is terminated, onionshare apparently does not notice
that. Would be better if onionshare would notice that. Is that a bug?

It seems like a "bug" in onionshare, or even the control port language
protocol itself since it (AFAICT) doesn't have a concept of the server
quitting mid-session. No signal is sent, and I haven't even found an
event one can explicitly subscribe to to learn when it shuts down. In
fact, any stem-application will, for instance, notice that Tor closed
its control port on the next send() or recv() on the socket, and then
throw a stem.SocketClosed.

Both nc and telnet will notice and automatically terminate as expected
once the filter was stopped. Therefore I guess the control protocol may
be sufficient.

Do you think this could be a bug in onionshare or onionshare? //cc
Damian, Micah

Cheers,
Patrick

[Damian Johnson]

Hi Patrick. Not sure I follow - a control port filter acts as a proxy...

Application => Filter => Tor Control Port

If the filter is shut down the socket the application is connected to
is... well, gone. Is the trouble that Onionshare gives a poor
indication when this happens? If so then indeed, that sounds like an
Onionshare bug.


On Mon, Nov 14, 2016 at 11:00 AM, Patrick Schleizer

Post by Patrick Schleizer

Post by anonym

Post by Patrick Schleizer
When the filter is terminated, onionshare apparently does not notice
that. Would be better if onionshare would notice that. Is that a bug?

It seems like a "bug" in onionshare, or even the control port language
protocol itself since it (AFAICT) doesn't have a concept of the server
quitting mid-session. No signal is sent, and I haven't even found an
event one can explicitly subscribe to to learn when it shuts down. In
fact, any stem-application will, for instance, notice that Tor closed
its control port on the next send() or recv() on the socket, and then
throw a stem.SocketClosed.

Both nc and telnet will notice and automatically terminate as expected
once the filter was stopped. Therefore I guess the control protocol may
be sufficient.
Do you think this could be a bug in onionshare or onionshare? //cc
Damian, Micah
Cheers,
Patrick

[anonym]

Post by Damian Johnson
Hi Patrick. Not sure I follow - a control port filter acts as a proxy...
Application => Filter => Tor Control Port

Exactly.

Post by Damian Johnson
If the filter is shut down the socket the application is connected to
is... well, gone. Is the trouble that Onionshare gives a poor
indication when this happens? If so then indeed, that sounds like an
Onionshare bug.

[...] any stem-application will, for instance, notice that Tor closed
its control port on the next send() or recv() on the socket, and then
throw a stem.SocketClosed.

i.e. it will not immediately notice that Tor has closed the socket. I
don't see why it should be any different when a stem-application is
connecting to the filter instead. I'm not sure why this is the case, but
my bet would be that it's related to how stem does async IO.

Cheers!

[Damian Johnson]

Post by anonym

[...] any stem-application will, for instance, notice that Tor closed
its control port on the next send() or recv() on the socket, and then
throw a stem.SocketClosed.

i.e. it will not immediately notice that Tor has closed the socket. I
don't see why it should be any different when a stem-application is
connecting to the filter instead. I'm not sure why this is the case, but
my bet would be that it's related to how stem does async IO.

Yes and no. Stem doesn't know that the socket is closed until it tries
to use it, *but* Stem is always listening to the socket. So Stem knows
right away when the socket is closed.

https://stem.torproject.org/api/socket.html#stem.socket.ControlSocket.is_alive

OnionShare can call the is_alive() method to determine if the
connection is still around or not. Could the filter perhaps be failing
to call close() on its sockets? Shouldn't matter (assuming it's python
GC should be taking care of that), but good practice to be sure.

[Patrick Schleizer]

Post by anonym

Post by Patrick Schleizer
About the packaging. If you like the genmkfile way to package things, I
could also do the packaging. Only disadvantage would be an extra
dependency on genmkfile.
https://github.com/Whonix/control-port-filter-python
https://github.com/Whonix/genmkfile
That would be trivial on my side since Tails control filter seems very
similar to control-port-filter-python. (control-port-filter-python
packaged with genmkfile is a lintian --pedantic warning free and to my
knowledge, fully complaint Debian source and binary package.) Otherwise,
I'd just wait for you.

Sure! I bet you'll beat me to it, so I don't want to be the blocker, but
perhaps I'll take over the packaging in the future, depending on how
things go (would this make sense to include in Debian?). However, first

Not sure if upload to Debian would benefit any of our projects or if
anyone besides our two projects would be interested in that package. Due
to release cycles, I guess we'd end up not using that package in
Tails/Whonix anyhow. I must admit, that this chance to get genmkfile
into Debian as a byproduct is more appealing than the filter itself.

If any other of Whonix's packages seems useful to be uploaded to Debian
or otherwise useful for Tails, I'd be happy to polish them if there are
any requests to return the favor and to improve the cooperation.

Post by anonym
1. A name that doesn't including 'tor'. What about 'grater' or
'onion-grater'? My inspiration is something like [0]. :) Otherwise
there's 'onion-filter' I guess.

What about control-port-filter-python? :)

I was rethinking this. The packaging may not be that important on the
Whonix side. Since the tails filter is small, clean, nice and simple and
"quite similar" to currently used control-port-filter-python... I could
just copy over the tails filter to control-port-filter-python
usr/sbin/cpfpd. (Along with some minor things such as debian/control
migrating dependencies list and systemd unit file differences.)

Post by anonym
2. A dedicated Git repo (trivial once we have a name).

https://github.com/Whonix/control-port-filter-python ? :)

The separate upstream tails filter repository would just help me to stay
on top of changes rather than following changes in the huge Tails main
repository. It's not very important either. If you'd ever find a
security critical issue, please notify me. Otherwise for newer features,
I'll probably learn about them early enough.

Post by anonym

Post by Patrick Schleizer
I added 'Flags=DiscardPK', which works and I thought that is useful at
least in case of Whonix. The workstation does not need to learn the
hidden service key since onionshare does not use it anyhow. Not sure
this (and the following) makes also sense in Tails?
(azazazazazazaz10 is the HS)
650 HS_DESC CREATED azazazazazazaz10 UNKNOWN UNKNOWN gliberrish REPLICA=0
650 HS_DESC CREATED azazazazazazaz10 UNKNOWN UNKNOWN gliberrish REPLICA=1
650 HS_DESC UPLOAD azazazazazazaz10 UNKNOWN gliberrish gliberrish
650 HS_DESC UPLOADED azazazazazazaz10 UNKNOWN gliberrish
650 HS_DESC CREATED azazazazazazaz10 UNKNOWN UNKNOWN dedacted REPLICA=0
650 HS_DESC CREATED azazazazazazaz10 UNKNOWN UNKNOWN dedacted REPLICA=1
650 HS_DESC UPLOAD azazazazazazaz10 UNKNOWN dedacted dedacted
650 HS_DESC UPLOADED azazazazazazaz10 UNKNOWN dedacted
I am not sure stem would complain about this, but I guess not

onionshare uses stem's `create_ephemeral_hidden_service()` convenience
method with `await_publication=True`, and it tracks the state of the
hidden services publication via HS_DESC events, so you have to be
careful here (e.g. only allowing the UPLOADED won't work, probably).
So, to me it looks like you need the following (untested) config,
although you might want to study the spec to see which events are to be
- pattern: '650 HS_DESC CREATED (\S+) (\S+) (\S+) \S+ (.+)'
replacement: '650 HS_DESC CREATED {} {} {} redacted {}'
- pattern: '650 HS_DESC UPLOAD (\S+) (\S+) .*'
replacement: '650 HS_DESC UPLOAD {} {} redacted redacted'
- pattern: '650 HS_DESC UPLOADED (\S+) (\S+) .+'
replacement: '650 HS_DESC UPLOADED {} {} redacted'
- pattern: '.*'
replacement: ''

That unfortunately does not work.

***@host:~/tcpfp$ ./tor-controlport-filter --listen-address 0.0.0.0 --debug
Tor control port filter started, listening on 0.0.0.0:9051
10.137.11.80:51904 (filter: whonix) connected: loaded filter: whonix
Final rules:
commands:
ADD_ONION:
- {pattern: 'NEW:BEST Port=80,(176[0-5][0-9])', replacement: 'NEW:BEST
Port=80,{client-address}:{}
Flags=DiscardPK'}
DEL_ONION:
- {pattern: .+}
GETCONF:
- {pattern: (__owningcontrollerprocess)}
GETINFO:
- {pattern: circuit-established}
- {pattern: status/circuit-established}
- pattern: net/listeners/socks
response:
- {pattern: 250-net/listeners/socks=".*", replacement:
'250-net/listeners/socks="127.0.0.1:9150"'}
- {pattern: version}
- {pattern: onions/current}
SIGNAL:
- {pattern: NEWNYM}
events:
CONF_CHANGED: {suppress: true}
HS_DESC:
- {pattern: 650 HS_DESC CREATED (\S+) (\S+) (\S+) \S+ (.+),
replacement: '650 HS_DESC
CREATED {} {} {} redacted {}'}
- {pattern: 650 HS_DESC UPLOAD (\S+) (\S+) .*, replacement: '650
HS_DESC UPLOAD
{} {} redacted redacted'}
- {pattern: 650 HS_DESC UPLOADED (\S+) (\S+) .+, replacement: '650
HS_DESC UPLOADED
{} {} redacted'}
- {pattern: .*, replacement: ''}
SIGNAL: {suppress: true}
restrict-stream-events: false

10.137.11.80:51904 (filter: whonix): -> PROTOCOLINFO 1
10.137.11.80:51904 (filter: whonix): <- 250-PROTOCOLINFO 1
10.137.11.80:51904 (filter: whonix): <- 250-AUTH METHODS=NULL
10.137.11.80:51904 (filter: whonix): <- 250-VERSION Tor="0.2.8.9
(git-cabd4ef300c6b3d6)"
10.137.11.80:51904 (filter: whonix): <- 250 OK
10.137.11.80:51904 (filter: whonix): -> AUTHENTICATE
10.137.11.80:51904 (filter: whonix): <- 250 OK
10.137.11.80:51904 (filter: whonix): -> SETEVENTS SIGNAL CONF_CHANGED
10.137.11.80:51904 (filter: whonix): suppressed subscription to event
'SIGNAL'
10.137.11.80:51904 (filter: whonix): suppressed subscription to event
'CONF_CHANGED'
10.137.11.80:51904 (filter: whonix): <- 250 OK
10.137.11.80:51904 (filter: whonix): -> GETCONF __owningcontrollerprocess
10.137.11.80:51904 (filter: whonix): <- 250 __OwningControllerProcess
10.137.11.80:51904 (filter: whonix): -> GETINFO version
10.137.11.80:51904 (filter: whonix): <- (multi-line)
250-version=0.2.8.9 (git-cabd4ef300c6b3d6)
250 OK
10.137.11.80:51904 (filter: whonix): -> SETEVENTS HS_DESC SIGNAL
CONF_CHANGED
10.137.11.80:51904 (filter: whonix) disconnected: client quit
----------------------------------------
Exception happened during processing of request from ('10.137.11.80', 51904)
Traceback (most recent call last):
File "/usr/lib/python3.4/socketserver.py", line 613, in
process_request_thread
self.finish_request(request, client_address)
File "/usr/lib/python3.4/socketserver.py", line 344, in finish_request
self.RequestHandlerClass(request, client_address, self)
File "/usr/lib/python3.4/socketserver.py", line 669, in __init__
self.handle()
File "./tor-controlport-filter", line 574, in handle
restrict_stream_events
File "./tor-controlport-filter", line 456, in handle_controlport_session
update_event_subscriptions(events)
File "./tor-controlport-filter", line 393, in update_event_subscriptions
if not rule.get('suppress', False) or \
AttributeError: 'list' object has no attribute 'get'
----------------------------------------

Post by anonym
However, I saw something in the stem code that might indicate it
actually uses the value you want to redact, but I might be mistaken.

I would not be too surprised if padding needs to be added. I.e. fake
values need to have compatible string lengths. Perhaps stem checks if
the string lengths make sense or stores these values for further use. I
however don't think it "really" needs them for onionshare functionality
so fake values may work. Will check once the config is working.

Cheers,
Patrick

[Patrick Schleizer]

Post by Patrick Schleizer

Post by anonym

Post by Patrick Schleizer
About the packaging. If you like the genmkfile way to package things, I
could also do the packaging. Only disadvantage would be an extra
dependency on genmkfile.
https://github.com/Whonix/control-port-filter-python
https://github.com/Whonix/genmkfile
That would be trivial on my side since Tails control filter seems very
similar to control-port-filter-python. (control-port-filter-python
packaged with genmkfile is a lintian --pedantic warning free and to my
knowledge, fully complaint Debian source and binary package.) Otherwise,
I'd just wait for you.

Sure! I bet you'll beat me to it, so I don't want to be the blocker, but
perhaps I'll take over the packaging in the future, depending on how
things go (would this make sense to include in Debian?). However, first

Not sure if upload to Debian would benefit any of our projects or if
anyone besides our two projects would be interested in that package. Due
to release cycles, I guess we'd end up not using that package in
Tails/Whonix anyhow. I must admit, that this chance to get genmkfile
into Debian as a byproduct is more appealing than the filter itself.
If any other of Whonix's packages seems useful to be uploaded to Debian
or otherwise useful for Tails, I'd be happy to polish them if there are
any requests to return the favor and to improve the cooperation.

Post by anonym
1. A name that doesn't including 'tor'. What about 'grater' or
'onion-grater'? My inspiration is something like [0]. :) Otherwise
there's 'onion-filter' I guess.

What about control-port-filter-python? :)
I was rethinking this. The packaging may not be that important on the
Whonix side. Since the tails filter is small, clean, nice and simple and
"quite similar" to currently used control-port-filter-python... I could
just copy over the tails filter to control-port-filter-python
usr/sbin/cpfpd. (Along with some minor things such as debian/control
migrating dependencies list and systemd unit file differences.)

Post by anonym
2. A dedicated Git repo (trivial once we have a name).

https://github.com/Whonix/control-port-filter-python ? :)
The separate upstream tails filter repository would just help me to stay
on top of changes rather than following changes in the huge Tails main
repository. It's not very important either. If you'd ever find a
security critical issue, please notify me. Otherwise for newer features,
I'll probably learn about them early enough.

Post by anonym

Post by Patrick Schleizer
I added 'Flags=DiscardPK', which works and I thought that is useful at
least in case of Whonix. The workstation does not need to learn the
hidden service key since onionshare does not use it anyhow. Not sure
this (and the following) makes also sense in Tails?
(azazazazazazaz10 is the HS)
650 HS_DESC CREATED azazazazazazaz10 UNKNOWN UNKNOWN gliberrish REPLICA=0
650 HS_DESC CREATED azazazazazazaz10 UNKNOWN UNKNOWN gliberrish REPLICA=1
650 HS_DESC UPLOAD azazazazazazaz10 UNKNOWN gliberrish gliberrish
650 HS_DESC UPLOADED azazazazazazaz10 UNKNOWN gliberrish
650 HS_DESC CREATED azazazazazazaz10 UNKNOWN UNKNOWN dedacted REPLICA=0
650 HS_DESC CREATED azazazazazazaz10 UNKNOWN UNKNOWN dedacted REPLICA=1
650 HS_DESC UPLOAD azazazazazazaz10 UNKNOWN dedacted dedacted
650 HS_DESC UPLOADED azazazazazazaz10 UNKNOWN dedacted
I am not sure stem would complain about this, but I guess not

onionshare uses stem's `create_ephemeral_hidden_service()` convenience
method with `await_publication=True`, and it tracks the state of the
hidden services publication via HS_DESC events, so you have to be
careful here (e.g. only allowing the UPLOADED won't work, probably).
So, to me it looks like you need the following (untested) config,
although you might want to study the spec to see which events are to be
- pattern: '650 HS_DESC CREATED (\S+) (\S+) (\S+) \S+ (.+)'
replacement: '650 HS_DESC CREATED {} {} {} redacted {}'
- pattern: '650 HS_DESC UPLOAD (\S+) (\S+) .*'
replacement: '650 HS_DESC UPLOAD {} {} redacted redacted'
- pattern: '650 HS_DESC UPLOADED (\S+) (\S+) .+'
replacement: '650 HS_DESC UPLOADED {} {} redacted'
- pattern: '.*'
replacement: ''

That unfortunately does not work.
Tor control port filter started, listening on 0.0.0.0:9051
10.137.11.80:51904 (filter: whonix) connected: loaded filter: whonix
- {pattern: 'NEW:BEST Port=80,(176[0-5][0-9])', replacement: 'NEW:BEST
Port=80,{client-address}:{}
Flags=DiscardPK'}
- {pattern: .+}
- {pattern: (__owningcontrollerprocess)}
- {pattern: circuit-established}
- {pattern: status/circuit-established}
- pattern: net/listeners/socks
'250-net/listeners/socks="127.0.0.1:9150"'}
- {pattern: version}
- {pattern: onions/current}
- {pattern: NEWNYM}
CONF_CHANGED: {suppress: true}
- {pattern: 650 HS_DESC CREATED (\S+) (\S+) (\S+) \S+ (.+),
replacement: '650 HS_DESC
CREATED {} {} {} redacted {}'}
- {pattern: 650 HS_DESC UPLOAD (\S+) (\S+) .*, replacement: '650
HS_DESC UPLOAD
{} {} redacted redacted'}
- {pattern: 650 HS_DESC UPLOADED (\S+) (\S+) .+, replacement: '650
HS_DESC UPLOADED
{} {} redacted'}
- {pattern: .*, replacement: ''}
SIGNAL: {suppress: true}
restrict-stream-events: false
10.137.11.80:51904 (filter: whonix): -> PROTOCOLINFO 1
10.137.11.80:51904 (filter: whonix): <- 250-PROTOCOLINFO 1
10.137.11.80:51904 (filter: whonix): <- 250-AUTH METHODS=NULL
10.137.11.80:51904 (filter: whonix): <- 250-VERSION Tor="0.2.8.9
(git-cabd4ef300c6b3d6)"
10.137.11.80:51904 (filter: whonix): <- 250 OK
10.137.11.80:51904 (filter: whonix): -> AUTHENTICATE
10.137.11.80:51904 (filter: whonix): <- 250 OK
10.137.11.80:51904 (filter: whonix): -> SETEVENTS SIGNAL CONF_CHANGED
10.137.11.80:51904 (filter: whonix): suppressed subscription to event
'SIGNAL'
10.137.11.80:51904 (filter: whonix): suppressed subscription to event
'CONF_CHANGED'
10.137.11.80:51904 (filter: whonix): <- 250 OK
10.137.11.80:51904 (filter: whonix): -> GETCONF __owningcontrollerprocess
10.137.11.80:51904 (filter: whonix): <- 250 __OwningControllerProcess
10.137.11.80:51904 (filter: whonix): -> GETINFO version
10.137.11.80:51904 (filter: whonix): <- (multi-line)
250-version=0.2.8.9 (git-cabd4ef300c6b3d6)
250 OK
10.137.11.80:51904 (filter: whonix): -> SETEVENTS HS_DESC SIGNAL
CONF_CHANGED
10.137.11.80:51904 (filter: whonix) disconnected: client quit
----------------------------------------
Exception happened during processing of request from ('10.137.11.80', 51904)
File "/usr/lib/python3.4/socketserver.py", line 613, in
process_request_thread
self.finish_request(request, client_address)
File "/usr/lib/python3.4/socketserver.py", line 344, in finish_request
self.RequestHandlerClass(request, client_address, self)
File "/usr/lib/python3.4/socketserver.py", line 669, in __init__
self.handle()
File "./tor-controlport-filter", line 574, in handle
restrict_stream_events
File "./tor-controlport-filter", line 456, in handle_controlport_session
update_event_subscriptions(events)
File "./tor-controlport-filter", line 393, in update_event_subscriptions
if not rule.get('suppress', False) or \
AttributeError: 'list' object has no attribute 'get'
----------------------------------------

This is now fixed:
https://phabricator.whonix.org/T574#10972

Best regards,
Patrick

[Patrick Schleizer]

Happy to report, that a few profiles have been successfully written.
That are using Whonix forked config parsing code.

They are now living here:

-
https://github.com/Whonix/control-port-filter-python/tree/master/usr/share/tor-controlport-filter/examples

There is one for onionshare, one for ricochet as well as one for
ZeroNet. So onionshare and ricochet will most likely run fine in the
next version of Whonix, Whonix 14.

I am impressed by the rewrite functionalities which are a blessing.
Ricochet does something rather ugly, requesting several GETINFO status
at once.

GETINFO status/circuit-established status/bootstrap-phase
net/listeners/socks

With Tails control port filter proxy, these are elegantly rewritten.

GETINFO:
- pattern: 'status/circuit-established status/bootstrap-phase
net/listeners/socks'
response:
- pattern: '250-status/bootstrap-phase=*'
replacement: '250-status/bootstrap-phase=NOTICE BOOTSTRAP
PROGRESS=100 TAG=done SUMMARY="Done"'
- pattern: '250-net/listeners/socks=".*"'
replacement: '250-net/listeners/socks="127.0.0.1:9150"'

The ZeroNet profile latter however might require some more hackery. Or
fixes in ZeroNet or fixes in the control port filter proxy. This is
probably because ZeroNet has custom code for Tor control protocol
authentication. Not using python-stem. ZeroNet works when having a
direct Tor control connection but not through the control port filter proxy.

- https://github.com/HelloZeroNet/ZeroNet/blob/master/src/Tor/TorManager.py

Reported two issues.

- https://github.com/HelloZeroNet/ZeroNet/issues/756
- https://github.com/HelloZeroNet/ZeroNet/issues/758

Best regards,
Patrick

[Patrick Schleizer]

Whonix has forked tor-controlport-filter by Tails.

https://github.com/Whonix/control-port-filter-python

Whonix is using a different configuration parser.

This is now documented in details here:

https://www.whonix.org/wiki/Dev/Control_Port_Filter_Proxy/tor-controlport-filter/config

Best regards,
Patrick

[anonym]

Post by Patrick Schleizer
Whonix has forked tor-controlport-filter by Tails.
https://github.com/Whonix/control-port-filter-python
Whonix is using a different configuration parser.

Yay! Let's try to make this fork short-lived! Note that Tails' version has changed quite a lot since you forked -- please try to keep your fork delta minimal (i.e. only do what *must* be done)!

Post by Patrick Schleizer
https://www.whonix.org/wiki/Dev/Control_Port_Filter_Proxy/tor-controlport-filter/config

So, since all your filters match *everything*, you could actually merge all the filters yourself into a single filter rule file, right? In fact, the only non-sytlistic reason you want merging of multiple matched rules is to allow Whonix users to add their own filters without overwriting yours (which causes issues during Whonix upgrades), correct?

If so, I think I have a simpler solution: we introduce a new top-level key called `override` which takes a list of strings. If such a filter is matched together with a filter whose name is listed in `override`, then the `override`-filter is merged into the other one, and has priority whenever there's any ambiguity. So with this, Whonix would ship a single, nicely commented filter called 'whonix', and users that need to modify it simply creates a filter with `override: ['whonix']`.

The only drawback I can see is that Whonix cannot organize the filters in separate files this way, but I think that is a small prize to pay for such a simple solution to this problem (which I fear otherwise can make it really hard to reason about filters when they grow in number). I actually think Whonix having a single file is slightly advantageous since you use the same match rules in every filter.

Would this work for you?

Beyond that, I'll add your `--listen-interface` option, and I'll add a `--filters-dir` option that can be used to override the default filters dir, and that can be passed multiple times (so you'll pass `--filters-dir /etc/tor-controlport-filter.d --filters-dir /usr/local/etc/tor-controlport-filter.d/`).

Would this be enough to kill your fork?

Cheers!

[Patrick Schleizer]

Post by anonym
Yay! Let's try to make this fork short-lived!

Yes! :)

Post by anonym
Note that Tails' version has changed quite a lot since you forked --

please try to keep your fork delta minimal (i.e. only do what *must* be
done)!

Our diff of the filter is quite mergable, I guess. In summary:

- filters = yaml.safe_load(fh.read())
- #!/usr/bin/python3 -u so python exceptions end up in journal
- config parser
- --listen-interface (which you already said you want to add)

(And the packaging stuff is not all that difficult?)

We have a ticket for merging Tails' version into Whonix's version so we
get your enhancement and so the diff looks simpler. [1]

Post by anonym
So, since all your filters match *everything*, you could actually

merge all the filters yourself into a single filter rule file, right? In
fact, the only non-sytlistic reason you want merging of multiple matched
rules is to allow Whonix users to add their own filters without
overwriting yours (which causes issues during Whonix upgrades), correct?

Post by anonym
If so, I think I have a simpler solution: we introduce a new top-level

key called `override` which takes a list of strings. If such a filter is
matched together with a filter whose name is listed in `override`, then
the `override`-filter is merged into the other one, and has priority
whenever there's any ambiguity. So with this, Whonix would ship a
single, nicely commented filter called 'whonix', and users that need to
modify it simply creates a filter with `override: ['whonix']`.

Post by anonym
The only drawback I can see is that Whonix cannot organize the filters

in separate files this way, but I think that is a small prize to pay for
such a simple solution to this problem (which I fear otherwise can make
it really hard to reason about filters when they grow in number). I
actually think Whonix having a single file is slightly advantageous
since you use the same match rules in every filter.

Post by anonym
Would this work for you?

No. (However, I am not sure I fully understood your proposal.
Technically at the moment all config file snippets are merged into the
same merged filter rule.)

Having the filters split in multiple files is very much desirable. We
want to ship a maximally restrictive config by default. And then allow
users to make it more permissive as required depending on their use case.

Just two different configs, one for maximum restrictive vs one for
custom maximum permissive is probably not right balance here.

(We match everything because of necessity - to keep the code complexity
manageable. Matching by IP would be a nice feature, so would be a
profile on the gateway that only matches `arm`.)

For example, adding onionshare is one of the safer applications one can
whitelist due to the limited range of listening ports it opens. ricochet
is more crazy, since all ports need to be allowed. (Something being
discussed with upstream so this may change for Debian buster.) Some
applications want to see the onion's private key [ricochet, so it can
restore it later], for others we can inject DiscardPK and redact it. I
think other applications will require access to further Tor internal
states. Ricochet uses STATUS_CLIENT, but fortunately also does not break
without it yet.

Having to use something like "override: ['whonix']" and somehow have a
hierarchy of files named that way, if they are splittable/stackable,
would of course not be an issue.

Post by anonym
Beyond that, I'll add your `--listen-interface` option, and I'll add a

`--filters-dir` option that can be used to override the default filters
dir, and that can be passed multiple times (so you'll pass
`--filters-dir /etc/tor-controlport-filter.d --filters-dir
/usr/local/etc/tor-controlport-filter.d/`).

That's great! (Please don't fail if the folder does not exist - logging
that only would be a lot better.)

Cheers,
Patrick

[1] https://phabricator.whonix.org/T612

[anonym]

Post by Patrick Schleizer

Post by anonym
Yay! Let's try to make this fork short-lived!

Yes! :)

Post by anonym
Note that Tails' version has changed quite a lot since you forked --

please try to keep your fork delta minimal (i.e. only do what *must* be
done)!
- filters = yaml.safe_load(fh.read())
- #!/usr/bin/python3 -u so python exceptions end up in journal
- config parser
- --listen-interface (which you already said you want to add)
(And the packaging stuff is not all that difficult?)
We have a ticket for merging Tails' version into Whonix's version so we
get your enhancement and so the diff looks simpler. [1]

Let me know when that has happened!

Post by Patrick Schleizer

Post by anonym
So, since all your filters match *everything*, you could actually
merge all the filters yourself into a single filter rule file, right? In
fact, the only non-sytlistic reason you want merging of multiple matched
rules is to allow Whonix users to add their own filters without
overwriting yours (which causes issues during Whonix upgrades), correct?
If so, I think I have a simpler solution: we introduce a new top-level
key called `override` which takes a list of strings. If such a filter is
matched together with a filter whose name is listed in `override`, then
the `override`-filter is merged into the other one, and has priority
whenever there's any ambiguity. So with this, Whonix would ship a
single, nicely commented filter called 'whonix', and users that need to
modify it simply creates a filter with `override: ['whonix']`.
The only drawback I can see is that Whonix cannot organize the filters
in separate files this way, but I think that is a small prize to pay for
such a simple solution to this problem (which I fear otherwise can make
it really hard to reason about filters when they grow in number). I
actually think Whonix having a single file is slightly advantageous
since you use the same match rules in every filter.
Would this work for you?

No. (However, I am not sure I fully understood your proposal.
Technically at the moment all config file snippets are merged into the
same merged filter rule.)

Yes, currently the multiple filters you use are merged into a single large filter. Hence, you might as well ship that merged filter (in a single file) instead of these multiple ones.

Post by Patrick Schleizer
Having the filters split in multiple files is very much desirable.

For now I'll ignore "desirability" due to organizational improvement -- let's only talk about the results.

Post by Patrick Schleizer
We
want to ship a maximally restrictive config by default. And then allow
users to make it more permissive as required depending on their use case.

Ack. So your default "maximally restrictive config" would be the single filter named 'whonix'. And users can create a filter with `override: ['whonis']` and their additional rules.

Post by Patrick Schleizer
Just two different configs, one for maximum restrictive vs one for
custom maximum permissive is probably not right balance here.

What balance is lost here?

Post by Patrick Schleizer
Having to use something like "override: ['whonix']" and somehow have a
hierarchy of files named that way, if they are splittable/stackable,
would of course not be an issue.

A single `override` filter can achieve the same as multiple `override` filters -- just manually merge them, and then you have that "single" `override` filter.

So, in the end, it seems that nothing in terms of expressibility is lost by not having general merging, but just the `override` functionality. Am I wrong, or is there still something unclear about this?

Post by Patrick Schleizer

Post by anonym
Beyond that, I'll add your `--listen-interface` option, and I'll add a

`--filters-dir` option that can be used to override the default filters
dir, and that can be passed multiple times (so you'll pass
`--filters-dir /etc/tor-controlport-filter.d --filters-dir
/usr/local/etc/tor-controlport-filter.d/`).
That's great! (Please don't fail if the folder does not exist - logging
that only would be a lot better.)

Noted!

Cheers!

[Patrick Schleizer]

Hi!

[override] will probably work for Whonix. Joy and me drafted a plan.

In one sentence: We at Whonix invent a new a separate config folder,
parse it with a yml merger python script, and generate another yml file
that gets passed to tor-controlport-filter by Tails.

In more detail:

- We'll at Whonix invent /usr/lib/tor-controlport-filter-merger.
- And ship that as opt-in or in a separate package by Whonix.
- (If opt-in, we enable it in a separate Whonix package.)

- /etc/tor-controlport-filter.d
-- We tell Whonix users to ignore it.
-- Internally used by /usr/lib/tor-controlport-filter .
-- Will contain
--- tails-default-profies.yml (for the sake of sharing the package and
perhaps we also benefit from a profile for arm/nyx)
--- 30_autogenerated.yml

- /etc/tor-controlport-filter-merger.d
-- Will be used by Whonix and its users
-- 30_whonix_default.yml - will by shipped by Whonix by default
-- 40_onionshare.yml - user defined
-- 40_ricochet.yml - another user defined etc.

- /usr/lib/tor-controlport-filter-merger parses both,
-- /etc/tor-controlport-filter-merger.d and
-- /usr/local/etc/tor-controlport-filter-merger.d (for Qubes-Whonix)
-- and creates /etc/tor-controlport-filter.d/30_autogenerated.yml

- Our tor-controlport-filter.service systemd service will in essence
look like this.
-- ExecStartPre=/usr/lib/tor-controlport-filter-merger
-- ExecStart=/usr/lib/tor-controlport-filter

Does that sound like that could work out?

Best regards,
Patrick

[anonym]

Post by Patrick Schleizer
Hi!
[override] will probably work for Whonix. Joy and me drafted a plan.
In one sentence: We at Whonix invent a new a separate config folder,
parse it with a yml merger python script, and generate another yml file
that gets passed to tor-controlport-filter by Tails.

Ok. My understanding of this proposal is that you no longer need any sort of "filter rules merging" in tor-controlport-filter itself, correct? If so, great! :) Feel free to send a PR with your other changes applied to tor-controlport-filter in Tails Git! Otherwise I'll do it myself later this week.

Post by Patrick Schleizer
- We'll at Whonix invent /usr/lib/tor-controlport-filter-merger.
- And ship that as opt-in or in a separate package by Whonix.
- (If opt-in, we enable it in a separate Whonix package.)
- /etc/tor-controlport-filter.d
-- We tell Whonix users to ignore it.
-- Internally used by /usr/lib/tor-controlport-filter .
-- Will contain
--- tails-default-profies.yml (for the sake of sharing the package

But they are not useful in Whonix since they only work for loopback connections (i.e. only for applications running on the gateway, which should be nothing except for tor, essentially). Right?

Post by Patrick Schleizer
and perhaps we also benefit from a profile for arm/nyx)

For the record, we'll remove Nyx/arm in Tails 2.10, due tomorrow. :)

Post by Patrick Schleizer
--- 30_autogenerated.yml
- /etc/tor-controlport-filter-merger.d
-- Will be used by Whonix and its users
-- 30_whonix_default.yml - will by shipped by Whonix by default
-- 40_onionshare.yml - user defined
-- 40_ricochet.yml - another user defined etc.
- /usr/lib/tor-controlport-filter-merger parses both,
-- /etc/tor-controlport-filter-merger.d and
-- /usr/local/etc/tor-controlport-filter-merger.d (for Qubes-Whonix)
-- and creates /etc/tor-controlport-filter.d/30_autogenerated.yml
- Our tor-controlport-filter.service systemd service will in essence
look like this.
-- ExecStartPre=/usr/lib/tor-controlport-filter-merger
-- ExecStart=/usr/lib/tor-controlport-filter
Does that sound like that could work out?

Yup, I don't see why this wouldn't work.

Cheers!

[Patrick Schleizer]

Post by anonym

Post by Patrick Schleizer
[override] will probably work for Whonix. Joy and me drafted a plan.
In one sentence: We at Whonix invent a new a separate config
folder, parse it with a yml merger python script, and generate
another yml file that gets passed to tor-controlport-filter by
Tails.

Ok. My understanding of this proposal is that you no longer need any
sort of "filter rules merging" in tor-controlport-filter itself,
correct? If so, great! :)

I guess so, right.

Unless any of the Tails profiles use '*'? But in that case we might be
able to just config-package-dev displace the profile.

Post by anonym
Feel free to send a PR with your other
changes applied to tor-controlport-filter in Tails Git!
Otherwise
I'll do it myself later this week.

Let's see who is faster. Can't say yet.

Post by anonym

Post by Patrick Schleizer
- We'll at Whonix invent /usr/lib/tor-controlport-filter-merger. -
And ship that as opt-in or in a separate package by Whonix. - (If
opt-in, we enable it in a separate Whonix package.)
- /etc/tor-controlport-filter.d -- We tell Whonix users to ignore
it. -- Internally used by /usr/lib/tor-controlport-filter . -- Will
contain --- tails-default-profies.yml (for the sake of sharing the
package

But they are not useful in Whonix since they only work for loopback
connections (i.e. only for applications running on the gateway, which
should be nothing except for tor, essentially). Right?

Right. [And a rather minor point...: tor-arm [now nyx] is one that could
use a profile. Users tend to create screenshots of arm, so redacting any
IP addresses would be nice. Also terminal emulators such as konsole
might have bugs. By limiting what what tor-arm gets to see it might
prevent exploiting a bug in the terminal emulator. So hypothetically
speaking, you have a profile for tor-arm, we would probably use it as well.]

Post by anonym

Post by Patrick Schleizer
and perhaps we also benefit from a profile for arm/nyx)

For the record, we'll remove Nyx/arm in Tails 2.10, due tomorrow. :)

Ah, I see. :)

Post by anonym

Post by Patrick Schleizer
--- 30_autogenerated.yml
- /etc/tor-controlport-filter-merger.d -- Will be used by Whonix
and its users -- 30_whonix_default.yml - will by shipped by Whonix
by default -- 40_onionshare.yml - user defined -- 40_ricochet.yml -
another user defined etc.
- /usr/lib/tor-controlport-filter-merger parses both, --
/etc/tor-controlport-filter-merger.d and --
/usr/local/etc/tor-controlport-filter-merger.d (for Qubes-Whonix)
-- and creates /etc/tor-controlport-filter.d/30_autogenerated.yml
- Our tor-controlport-filter.service systemd service will in
essence look like this. --
ExecStartPre=/usr/lib/tor-controlport-filter-merger --
ExecStart=/usr/lib/tor-controlport-filter
Does that sound like that could work out?

Yup, I don't see why this wouldn't work.

Awesome!

Cheers,
Patrick

[anonym]

Post by Patrick Schleizer

Post by anonym

Post by Patrick Schleizer
[override] will probably work for Whonix. Joy and me drafted a plan.
In one sentence: We at Whonix invent a new a separate config
folder, parse it with a yml merger python script, and generate
another yml file that gets passed to tor-controlport-filter by
Tails.

Ok. My understanding of this proposal is that you no longer need any
sort of "filter rules merging" in tor-controlport-filter itself,
correct? If so, great! :)

I guess so, right.
Unless any of the Tails profiles use '*'? But in that case we might be
able to just config-package-dev displace the profile.

Tails doesn't use `hosts` (previously, `match-hosts` -- the `match-` prefix has been dropped for all three matching rules) but only `exe-paths` and `users`, and only with static, glob-less patterns.

Post by Patrick Schleizer

Post by anonym
Feel free to send a PR with your other
changes applied to tor-controlport-filter in Tails Git!
Otherwise
I'll do it myself later this week.

Let's see who is faster. Can't say yet.

Seems I was. :)

Post by Patrick Schleizer

Post by anonym

Post by Patrick Schleizer
- /etc/tor-controlport-filter.d -- We tell Whonix users to ignore
it. -- Internally used by /usr/lib/tor-controlport-filter . -- Will
contain --- tails-default-profies.yml (for the sake of sharing the
package

But they are not useful in Whonix since they only work for loopback
connections (i.e. only for applications running on the gateway, which
should be nothing except for tor, essentially). Right?

Right. [And a rather minor point...: tor-arm [now nyx] is one that could
use a profile. Users tend to create screenshots of arm, so redacting any
IP addresses would be nice. Also terminal emulators such as konsole
might have bugs. By limiting what what tor-arm gets to see it might
prevent exploiting a bug in the terminal emulator. So hypothetically
speaking, you have a profile for tor-arm, we would probably use it as well.]

Sure, but we won't. I expect that a profile very similar to the one we have for Onion Circuits would do if you just want to use it as a circuit/stream viewer.

Cheers!

[Patrick Schleizer]

Hello anonym!

Post by anonym
Feel free to send a PR with your other
changes applied to tor-controlport-filter in Tails Git!
Otherwise
I'll do it myself later this week.

Joy rebased Whonix's changes on top of your new version.

base:
https://git-tails.immerda.ch/tails/plain/config/chroot_local-includes/usr/local/lib/tor-controlport-filter

fork:
https://github.com/joysn/control-port-filter-python/blob/master/usr/lib/tor-controlport-filter

The diff looks simple, I guess.

Please ignore:

- config parser changes

Please take:

- #!/usr/bin/python3 -u (makes eventual python exceptions and up in journal)
- Use yml.safe_load and Python exceptions in journalctl
- add --listen_interface option

One minor bug, I think, first line is supposed to be

#!/usr/bin/python3 -u
(not -v)

Best regards,
Patrick

[anonym]

Post by Patrick Schleizer
Hello anonym!

Post by anonym
Feel free to send a PR with your other
changes applied to tor-controlport-filter in Tails Git!
Otherwise
I'll do it myself later this week.

Joy rebased Whonix's changes on top of your new version.
https://git-tails.immerda.ch/tails/plain/config/chroot_local-includes/usr/local/lib/tor-controlport-filter
https://github.com/joysn/control-port-filter-python/blob/master/usr/lib/tor-controlport-filter
The diff looks simple, I guess.

If you see my email from earlier today, I already did this: https://mailman.boum.org/pipermail/tails-dev/2017-January/011190.html

Post by Patrick Schleizer
- config parser changes

I did!

However, in your repo I still see that commit bed6399b contains the merge_yml() code. You are gonna do that externally, right? However, the commit talks about "add /etc/tor-controlport-filter.d configuration support", so perhaps it was a mistake (i.e. you wanted to add a `--filter-dir` option, but picked the wrong commit)?

Post by Patrick Schleizer
- #!/usr/bin/python3 -u (makes eventual python exceptions and up in journal)
- Use yml.safe_load and Python exceptions in journalctl
- add --listen_interface option

These were the commits I imported.

Post by Patrick Schleizer
One minor bug, I think, first line is supposed to be
#!/usr/bin/python3 -u
(not -v)

That error was not present in the code I looked at. Strange.

Cheers!

[Patrick Schleizer]

Post by anonym

Post by Patrick Schleizer
One minor bug, I think, first line is supposed to be
#!/usr/bin/python3 -u (not -v)

That error was not present in the code I looked at. Strange.

Probably by the time you looked at it, Joy already fixed it. Anyhow.
It's looking alright now.

Post by anonym

Post by Patrick Schleizer
- #!/usr/bin/python3 -u (makes eventual python exceptions and up in
journal) - Use yml.safe_load and Python exceptions in journalctl -
add --listen_interface option

These were the commits I imported.

Great!

Post by anonym

Post by Patrick Schleizer
Hello anonym!

Post by anonym
Feel free to send a PR with your other changes applied to
tor-controlport-filter in Tails Git! Otherwise I'll do it myself
later this week.

Joy rebased Whonix's changes on top of your new version.
https://git-tails.immerda.ch/tails/plain/config/chroot_local-includes/usr/local/lib/tor-controlport-filter
https://github.com/joysn/control-port-filter-python/blob/master/usr/lib/tor-controlport-filter

The diff looks simple, I guess.

Post by anonym
https://mailman.boum.org/pipermail/tails-dev/2017-January/011190.html

Post by Patrick Schleizer
- config parser changes

I did!
However, in your repo I still see that commit bed6399b contains the
merge_yml() code. You are gonna do that externally, right? However,
the commit talks about "add /etc/tor-controlport-filter.d
configuration support", so perhaps it was a mistake (i.e. you wanted
to add a `--filter-dir` option, but picked the wrong commit)?

Created https://phabricator.whonix.org/T617 for it.

What's next? What else? :)

Can we implement /usr/lib/tor-controlport-filter-merger directly in
https://github.com/Whonix/control-port-filter-python or would you prefer
if we implement that elsewhere? Should we implement the systemd override
to actually use it in Whonix elsewhere? If done right, if we move the
Whonix config into another Whonix package, it would not interfere with
Tails.

I'll take
https://git-tails.immerda.ch/tails/tree/config/chroot_local-includes/etc/tor-controlport-filter.d?h=feature/12173-end-whonix-controlport-filter-fork
and add to https://github.com/Whonix/control-port-filter-python ? Then
you should be able to build and install that package on Tails if you wish.

(lintian --pedantic warning free as well as most likely reproducible on
stretch.)

Cheers,
Patrick

[Patrick Schleizer]

Patch by Joy. Otherwise it does not work for us. Do you think you could
merge this patch?

https://github.com/joysn/control-port-filter-python/commit/6f488c14980e8b5c58a42374649c4d5725c8296e#diff-7414879ce81f5586d790820540d0ca05

Best regards,
Patrick

[anonym]

Post by Patrick Schleizer
Patch by Joy. Otherwise it does not work for us. Do you think you could
merge this patch?

No; the "match-"-prefix was intentionally dropped, so please `s/match-//g` in all your scripts and filter files.

Cheers!

[Patrick Schleizer]

Post by anonym

Post by Patrick Schleizer
Patch by Joy. Otherwise it does not work for us. Do you think you could
merge this patch?

No; the "match-"-prefix was intentionally dropped, so please `s/match-//g` in all your scripts and filter files.
Cheers!

Awesome, this is done in git master.

Best regards,
Patrick

[Patrick Schleizer]

Happy to report, that tor-controlport-filter learned sd_notify, now got
support for systemd's watchdog feature. Using python3-sdnotify from
packages.debian.org.

To be found in git master. Git commits, test results can be found here.

https://phabricator.whonix.org/T274#12423

https://github.com/Whonix/control-port-filter-python

Cheers,
Patrick

[Patrick Schleizer]

Hi!

Please reviewer and merge into onion-grater.

https://github.com/adrelanos/onion-grater-remote.git

branch:
sd-notify

https://github.com/adrelanos/onion-grater-remote/tree/sd-notify

Cheers,
Patrick

[Patrick Schleizer]

Hi!

https://github.com/Whonix/onion-grater/commit/70e735dae1c15920c356b07fc6aaf4b9589b465a

Please review and merge.

The more I think about it, perhaps we could abolish DEFAULT_COOKIE_PATH
= '/run/tor/control.authcookie' altogether?

PROTOCOLINFO tells controllers (like stem) where the cookie file is
located. It doesn't rely on a default path. For an example of what this
looks like on the wire please see...

https://stem.torproject.org/faq.html#i-m-using-cookie-authentication

Then we might be able to simply this and just use authenticate().

Cheers,
Patrick

[intrigeri]

Hi,

Post by Patrick Schleizer
https://github.com/Whonix/onion-grater/commit/70e735dae1c15920c356b07fc6aaf4b9589b465a
Please review and merge.

Code looks OK, quick manual testing in Tails works fine. I'll wait for
my full Tails test suite run to confirm this and then I expect I'll
push this to tails.git today :)

Post by Patrick Schleizer
The more I think about it, perhaps we could abolish DEFAULT_COOKIE_PATH
= '/run/tor/control.authcookie' altogether?
[...]
Then we might be able to simply this and just use authenticate().

No strong opinion here, feel free to send a patch if you feel strongly
about it. Bonus points if the patch has been quickly tested in
a running Tails system.

Cheers,

[Patrick Schleizer]

Hi,

could you please add this trivial fix?

https://github.com/Whonix/control-port-filter-python/commit/30c1de54f9feaa26464842241e217be6edf3b464

(fixes txtorcon compatibility)

Cheers,
Patrick

[1] https://github.com/meejah/txtorcon/issues/215#issuecomment-290277209

[anonym]

Post by Patrick Schleizer
Hi,
could you please add this trivial fix?
https://github.com/Whonix/control-port-filter-python/commit/30c1de54f9feaa26464842241e217be6edf3b464

I found your onion-grater repo, but it is not based on the upstream repo, has a different directory layout full of Whonix- and Linux-specific stuff. How am I, as the upstream, suppose to work with this?

Cheers!

[anonym]

Post by anonym
Feel free to send a PR with your other changes applied to
tor-controlport-filter in Tails Git! Otherwise I'll do it myself
later this week.

Done in Tails' branch: feature/12173-end-whonix-controlport-filter-fork

I also opened https://labs.riseup.net/code/issues/12173 to track this.

Cheers!

[Patrick Schleizer]

https://git.tails.boum.org/tails/tree/config/chroot_local-includes/usr/local/lib/tor-controlport-filter?h=feature/7870-include_onionshare

When I visit that link, I cannot proceed.

Your connection is not secure
The owner of git.tails.boum.org has configured their website improperly. To protect your information from being stolen, Firefox has not connected to this website.
This site uses HTTP Strict Transport Security (HSTS) to specify that Firefox only connect to it securely. As a result, it is not possible to add an exception for this certificate.

Thanks anonym, for all your work on tails control port filter and
replies. Very exciting developments! I am preparing responses and will
test it soonish.

Cheers,
Patrick

[anonym]

Post by Patrick Schleizer

https://git.tails.boum.org/tails/tree/config/chroot_local-includes/usr/local/lib/tor-controlport-filter?h=feature/7870-include_onionshare

When I visit that link, I cannot proceed.

Your connection is not secure
The owner of git.tails.boum.org has configured their website improperly. To protect your information from being stolen, Firefox has not connected to this website.
This site uses HTTP Strict Transport Security (HSTS) to specify that Firefox only connect to it securely. As a result, it is not possible to add an exception for this certificate.

IIRC there's some experimentation with Let's Encrypt. You can of course
instead pull Tails Git, check out the feature/7870-include_onionshare
branch, and look at:

* config/chroot_local-includes/usr/local/lib/tor-controlport-filter
* config/chroot_local-includes/etc/tor-controlport-filter.d/

Post by Patrick Schleizer
Thanks anonym, for all your work on tails control port filter and
replies. Very exciting developments! I am preparing responses and will
test it soonish.

Yay! Perhaps you can add ***@secure-os.org to Cc as intrigeri
suggested?

Cheers!

[intrigeri]

Post by anonym

Post by Patrick Schleizer

https://git.tails.boum.org/tails/tree/config/chroot_local-includes/usr/local/lib/tor-controlport-filter?h=feature/7870-include_onionshare

When I visit that link, I cannot proceed.

[...]

We try to never advertise links to https://git.t.b.o for
this very reason. The correct link is:

https://git-tails.immerda.ch/tails/tree/config/chroot_local-includes/usr/local/lib/tor-controlport-filter?h=feature/7870-include_onionshare

Post by anonym
IIRC there's some experimentation with Let's Encrypt.

Not that I know of. Maybe you're confused by me deploying Let's
Encrypt on our infra, while our Git is hosted by immerda (so mostly
out of our control).

[Patrick Schleizer]

Post by anonym
https://tails.boum.org/news/report_2016_09/#index2h1
and look at the documentation at the top of the script, and the filter
rules we ship to get an idea of what it can do.
As you can see, in Tails we use match-exe-paths and match-users a lot,
but since you won't have access to these I guess you want something like
match-hosts, so that the filter is picked based on the client's
(non-localhost) address (= VM). Right?

The match-hosts works well for Whonix. Set to "*" for now.

Post by anonym

Post by Patrick Schleizer

- https://phabricator.whonix.org/T562

This is about parsing add_onion and whitelisting sane commands rather
than letting through everything.

For any command we allow a list of regexes for the arguments. If a
command doesn't match any of them, it is filtered.

Sounds good!

Post by anonym

Post by Patrick Schleizer
add_onion is not a whitelist/not whitelist.

I do not understand this sentence...

Originally we at Whonix thought we wanted "wildcard support". Meant,
matching just "add_onion *", i.e. letting everything through
that comes after "add_onion". As it turned out, that was insufficient.
For example, add_onion flag "nonanonymous" ought to be avoided.

Tails control port filter proxy seems to keep care of that.

Post by anonym

Post by Patrick Schleizer
Buggy applications or by user mistake, they could choose the add_onion
flag nonanonymous, which would be a disaster. We also don't know what
Tor control protocol upgrades are coming in the years to come. So I
strongly suggest a only letting through whitelisted syntaxes.

... but this I get and agree with. Currently we require ADD_ONION for
onionshare to have args matching the regex 'NEW:BEST Port=80,176\d\d'.

Great!

Post by anonym

Post by Patrick Schleizer
Malicious applications could make the Tor HS listener bind on the wrong
interface. In Whonix-Gateway, maliciously listen on Whonix-Gateway.
Which could be fatal if we had also a real Tor ControlPort open there.
Does that make sense? I am not sure it applies to Tails, that depends on
your design and threat model, but it is however an interesting thought
that can inspire to finding more security issues with it.
Also it may be worth making sure it can only bind to specified (and
configureable) local ports?

Ack, this is an issue. In Tails there must be an explicit firewall rule
allowing a user listening on some port, so I think we are covered.

Ok.

Post by anonym

Post by Patrick Schleizer
For connectivity, we need to remove 127.0.0.1 and replace it with
Whonix-Workstation IP. That is currently done with the following code
block that I was going to merge with T562.
https://github.com/Whonix/control-port-filter-python/blob/6a131266a8dc8f98ff22a3b83fae9d43e38b3127/usr/sbin/cpfpd#L345-L375

Got it. Our filter doesn't do this (as we do not have this need) but I
feel a general solution could be to allow sed-like rewriting rules, e.g.
- pattern: 'NEW:BEST Port=80,(176\d\d)'
replacement: 'NEW:BEST Port=80,10.137.6.41:{0}'
which would be easy to implement.

Yes, that would work for Whonix.

Post by anonym

Post by Patrick Schleizer

- https://phabricator.whonix.org/T564

Protecting cpfpy from DDOS from client applications. Not sure that
matters for Tails?

We do not do much specific here. What kind of DoS are you talking about
here? Eating up all RAM or crashing the filter via oom kill? Preventing
the filter from serving other clients? We admittedly do not do much here
except that each client is dealt with in a separate thread, and that
client requests are limited to 1024 bytes.

Yes, I meant crashing the filter or making the whole computer unusable
by flooding cpfpy with too much requests.

Post by anonym

Post by Patrick Schleizer
- Supports logging.

I'm not sure what type of logging you are talking about here. Currently
it uses print() (to stdout) and flush, which works well with the journal
when run as a systemd service.Normally only filtered commands are
logged, but there's also a --complain mode which logs all client
requests, which is useful when writing rules for a new application.

Sounds fine!

Post by anonym

Post by Patrick Schleizer
- systemd support

Not sure what this means. We have a .service file.

That's it.

Post by anonym

Post by Patrick Schleizer
- When request is 'getinfo net/listeners/socks' answer with a lie
'250-net/listeners/socks="127.0.0.1:9150"'.

Nope. Why is this needed?

Rather minor: Not revealing all the listeners to all connected
workstations.

Post by anonym
I could even imagine yet another rule-type for solving these types of
- pattern: 'net/listeners/socks'
respond: '250-net/listeners/socks="127.0.0.1:9150"'

Seems great!

[anonym]

Post by Patrick Schleizer

Post by anonym
https://tails.boum.org/news/report_2016_09/#index2h1
and look at the documentation at the top of the script, and the filter
rules we ship to get an idea of what it can do.
As you can see, in Tails we use match-exe-paths and match-users a lot,
but since you won't have access to these I guess you want something like
match-hosts, so that the filter is picked based on the client's
(non-localhost) address (= VM). Right?

The match-hosts works well for Whonix. Set to "*" for now.

See my other answer why you perhaps want to give this another quick thought.

Post by Patrick Schleizer

Post by anonym

Post by Patrick Schleizer
add_onion is not a whitelist/not whitelist.

I do not understand this sentence...

Originally we at Whonix thought we wanted "wildcard support". Meant,
matching just "add_onion *", i.e. letting everything through
that comes after "add_onion". As it turned out, that was insufficient.
For example, add_onion flag "nonanonymous" ought to be avoided.

Got it!

Post by Patrick Schleizer
Tails control port filter proxy seems to keep care of that.

Regexes > globs! :)

Post by Patrick Schleizer

Post by anonym

Post by Patrick Schleizer

- https://phabricator.whonix.org/T564

Protecting cpfpy from DDOS from client applications. Not sure that
matters for Tails?

We do not do much specific here. What kind of DoS are you talking about
here? Eating up all RAM or crashing the filter via oom kill? Preventing
the filter from serving other clients? We admittedly do not do much here
except that each client is dealt with in a separate thread, and that
client requests are limited to 1024 bytes.

Yes, I meant crashing the filter or making the whole computer unusable
by flooding cpfpy with too much requests.

So the scenario is something like this: an attacker compromises the
workstation, and then want to crash the filter running on the gateway
(or even crash the whole gateway)? If so, IMHO DoS is the least of our
worries since all gateway activity (= user activity) now is compromised.
I mean, the attacker can DoS the workstation by killing all user
processes or whatever.

... but now I get that you may run several workstations (in Qubes, I
guess), and then this makes sense.

Quick solution that a determined adversary probably easily can work
around: in the gatway's firewall rules, rate limit the traffic to the
control filter per workstation.

Slightly more involved, slightly more efficient solution: we could add
an option to the filter making the server forking instead of threading,
and then adjust OOM scores and NICEness for these subprocesses so they
are low prio for the CPU, and prone to get OOM killed if memory gets
low. The server process itself will be configured in the opposite way
(you can add the options via a systemd override).

Any way, DoS protection is pretty hard... and I doubt it's much of a
problem on *local* systems. Rather DoS gives away that a workstation has
been compromised, instead of it remaining in a stealthy surveillance
mode, which I think I must consider worse (even when limited to a single
workstation == application).

Post by Patrick Schleizer

Post by anonym
I could even imagine yet another rule-type for solving these types of
- pattern: 'net/listeners/socks'
respond: '250-net/listeners/socks="127.0.0.1:9150"'

Seems great!

In the end, this was generalised into:

GETINFO:
- pattern: 'net/listeners/socks'
response:
- pattern: '250-net/listeners/socks=".*"'
- replacement: '250-net/listeners/socks="127.0.0.1:9150"'

Cheers!

[Patrick Schleizer]

Post by anonym

Post by Patrick Schleizer

Post by anonym

Post by Patrick Schleizer

- https://phabricator.whonix.org/T564

Protecting cpfpy from DDOS from client applications. Not sure that
matters for Tails?

We do not do much specific here. What kind of DoS are you talking about
here? Eating up all RAM or crashing the filter via oom kill? Preventing
the filter from serving other clients? We admittedly do not do much here
except that each client is dealt with in a separate thread, and that
client requests are limited to 1024 bytes.

Yes, I meant crashing the filter or making the whole computer unusable
by flooding cpfpy with too much requests.

So the scenario is something like this: an attacker compromises the
workstation, and then want to crash the filter running on the gateway
(or even crash the whole gateway)?

Yes.

Post by anonym
If so, IMHO DoS is the least of our
worries since all gateway activity (= user activity) now is compromised.
I mean, the attacker can DoS the workstation by killing all user
processes or whatever.
... but now I get that you may run several workstations (in Qubes, I
guess), and then this makes sense.

Yes.

Post by anonym
Quick solution that a determined adversary probably easily can work
around: in the gatway's firewall rules, rate limit the traffic to the
control filter per workstation.

I will consider this, useful, but guess it's not a priority as you
pointed out.

Post by anonym
Slightly more involved, slightly more efficient solution: we could add
an option to the filter making the server forking instead of threading,
and then adjust OOM scores and NICEness for these subprocesses so they
are low prio for the CPU, and prone to get OOM killed if memory gets
low. The server process itself will be configured in the opposite way
(you can add the options via a systemd override).
Any way, DoS protection is pretty hard... and I doubt it's much of a
problem on *local* systems. Rather DoS gives away that a workstation has
been compromised, instead of it remaining in a stealthy surveillance
mode, which I think I must consider worse (even when limited to a single
workstation == application).

Post by Patrick Schleizer

Post by anonym
I could even imagine yet another rule-type for solving these types of
- pattern: 'net/listeners/socks'
respond: '250-net/listeners/socks="127.0.0.1:9150"'

Seems great!

- pattern: 'net/listeners/socks'
- pattern: '250-net/listeners/socks=".*"'
- replacement: '250-net/listeners/socks="127.0.0.1:9150"'

That crashes the filter for me.

config:

---
- match-exe-paths:
- '*'
match-users:
- '*'
match-hosts:
- '*'
commands:
SIGNAL:
- 'NEWNYM'
GETINFO:
- 'circuit-established'
GETINFO:
- pattern: 'net/listeners/socks'
response:
- pattern: '250-net/listeners/socks=".*"'
- replacement: '250-net/listeners/socks="127.0.0.1:9150"'

run:

***@host:~$ ./tor-controlport-filter --listen-address 0.0.0.0 --debug
Tor control port filter started, listening on 0.0.0.0:9051
10.137.11.80:49996 (filter: tor-browser) connected: loaded filter:
tor-browser
Final rules:
commands:
GETCONF:
- {pattern: ()}
GETINFO:
- pattern: net/listeners/socks
response:
- {pattern: 250-net/listeners/socks=".*"}
- {replacement: '250-net/listeners/socks="127.0.0.1:9150"'}
SIGNAL:
- {pattern: NEWNYM}
events: {}
restrict-stream-events: false

10.137.11.80:49996 (filter: tor-browser): -> GETINFO net/listeners/socks
10.137.11.80:49996 (filter: tor-browser) disconnected: client quit
----------------------------------------
Exception happened during processing of request from ('10.137.11.80', 49996)
Traceback (most recent call last):
File "/usr/lib/python3.4/socketserver.py", line 613, in
process_request_thread
self.finish_request(request, client_address)
File "/usr/lib/python3.4/socketserver.py", line 344, in finish_request
self.RequestHandlerClass(request, client_address, self)
File "/usr/lib/python3.4/socketserver.py", line 669, in __init__
self.handle()
File "./tor-controlport-filter", line 550, in handle
restrict_stream_events
File "./tor-controlport-filter", line 466, in handle_controlport_session
response_rewriter=response_rewriter)
File "./tor-controlport-filter", line 302, in proxy_line
new_response = response_rewriter(response)
File "./tor-controlport-filter", line 447, in _response_rewriter
return rewrite_matched_lines(rule['response'], lines)
File "./tor-controlport-filter", line 331, in rewrite_matched_lines
for line in split_lines]) + "\r\n"
File "./tor-controlport-filter", line 331, in <listcomp>
for line in split_lines]) + "\r\n"
File "./tor-controlport-filter", line 324, in rewrite_matched_line
return rewrite_line(replacers, line)
File "./tor-controlport-filter", line 319, in rewrite_line
return r['replacement'].format(*match.groups()) + terminator
KeyError: 'replacement'
----------------------------------------

Cheers,
Patrick

[anonym]

Post by Patrick Schleizer
That crashes the filter for me.

Argh, I meant:

GETINFO:
- pattern: 'net/listeners/socks'
response:
- pattern: '250-net/listeners/socks=".*"'
replacement: '250-net/listeners/socks="127.0.0.1:9150"'

(Notice the removed "-" in front of "replacement".)

[Patrick Schleizer]

Post by anonym

Post by Patrick Schleizer
That crashes the filter for me.

- pattern: 'net/listeners/socks'
- pattern: '250-net/listeners/socks=".*"'
replacement: '250-net/listeners/socks="127.0.0.1:9150"'
(Notice the removed "-" in front of "replacement".)

That works for me!

Cheers,
Patrick

[Patrick Schleizer]

Hi there,

sorry for the delay, I got side tracked with other stuff.

My first and summary impression is, that this is looking excellent!

./tor-controlport-filter --listen-address 9052
Tor control port filter started, listening on 9052:9051

Do you see any reason in Whonix not to use the following...?

match-hosts:
- '*'

What I found confusing is, that "SIGNAL NEWNYM" is allowed, but being
case sensitive, i.e. "signal newnym" being blocked.

What do you suggest Whonix should use to pass --listen-address? A system
drop-in file overwriting ExecStart?

Cheers,
Patrick

[anonym]

Post by Patrick Schleizer
Hi there,
sorry for the delay, I got side tracked with other stuff.
My first and summary impression is, that this is looking excellent!

\o/

Post by Patrick Schleizer
./tor-controlport-filter --listen-address 9052
Tor control port filter started, listening on 9052:9051
Do you see any reason in Whonix not to use the following...?
- '*'

Principle of least privilege and defense in depth, I guess. If your
threat model supports that any host with access to the gateway can use
the Tor control port, then it's fine. Otherwise, perhaps you solve it on
the firewall-level instead. But if a static address is used for the
workstation, and its the only expected client, then I think locking it
down is a good idea, especially when it is so cheap (just a static
configuration).

Post by Patrick Schleizer
What I found confusing is, that "SIGNAL NEWNYM" is allowed, but being
case sensitive, i.e. "signal newnym" being blocked.

The command ("SIGNAL") is not case sensitive (e.g. "signal NEWNYM" is
eq. to "SIGNAL NEWNYM") per the Tor control port specification, and the
filter knows this. For arguments it depends on the command, and for
simplicity the filter tries to understand as little as possible of the
underlying language, so the responsibility is on the author of the
config file. However, it's fairly easy to profile an application with
the --complain option so I'm not worried about this being an issue.

Post by Patrick Schleizer
What do you suggest Whonix should use to pass --listen-address? A system
drop-in file overwriting ExecStart?

Yes, an override like that seems like the way to go.

Cheers!

[Patrick Schleizer]

Post by anonym

Post by Patrick Schleizer
Hi there,
sorry for the delay, I got side tracked with other stuff.
My first and summary impression is, that this is looking excellent!

\o/

Post by Patrick Schleizer
./tor-controlport-filter --listen-address 9052
Tor control port filter started, listening on 9052:9051
Do you see any reason in Whonix not to use the following...?
- '*'

Principle of least privilege and defense in depth, I guess. If your
threat model supports that any host with access to the gateway can use
the Tor control port, then it's fine. Otherwise, perhaps you solve it on
the firewall-level instead. But if a static address is used for the
workstation, and its the only expected client, then I think locking it
down is a good idea, especially when it is so cheap (just a static
configuration).

We don't have static addresses in Qubes-Whonix yet. Will come in Qubes
4.0. Then indeed match-hosts will be a great feature for us.

Post by anonym

Post by Patrick Schleizer
What I found confusing is, that "SIGNAL NEWNYM" is allowed, but being
case sensitive, i.e. "signal newnym" being blocked.

The command ("SIGNAL") is not case sensitive (e.g. "signal NEWNYM" is
eq. to "SIGNAL NEWNYM") per the Tor control port specification, and the
filter knows this. For arguments it depends on the command, and for
simplicity the filter tries to understand as little as possible of the
underlying language, so the responsibility is on the author of the
config file. However, it's fairly easy to profile an application with
the --complain option so I'm not worried about this being an issue.

Okay.

Post by anonym

Post by Patrick Schleizer
What do you suggest Whonix should use to pass --listen-address? A system
drop-in file overwriting ExecStart?

Yes, an override like that seems like the way to go.

Alright. :)

Cheers,
Patrick

[Patrick Schleizer]

Hi,

it's now packaged and lintian pedantic clean. The package should be
generic (work in Whonix and Tails at the same time) for the most part.

The missing part is Tails' config files. Since I don't know if you want
to actually use that package, I skipped Tails' config files and just
dropped Whonix's config file.

If you like to use that package, I would be happy to add Tails' config
[remove the only Whonix specific thing, the config]. [And then add
Whonix's config to some other of Whonix's packages.]

https://github.com/Whonix/control-port-filter-python

Best regards,
Patrick

[Patrick Schleizer]

Noticed one incompatibility.

ZeroNet uses custom code rather than python-stem to talk to Tor control
protocol. It's line handling works with original Tor, but not with the
filter.

https://github.com/HelloZeroNet/ZeroNet/issues/756

https://github.com/Whonix/control-port-filter-python/blob/master/usr/share/tor-controlport-filter/examples/40_zeronet.yml

Best regards,
Patrick

[anonym]

Post by Patrick Schleizer
Noticed one incompatibility.
ZeroNet uses custom code rather than python-stem to talk to Tor control
protocol. It's line handling works with original Tor, but not with the
filter.

The filter *should* be able to deal with any client implementation as long as it follow the control-spec, but there can of course be bugs.

Post by Patrick Schleizer
https://github.com/HelloZeroNet/ZeroNet/issues/756
https://github.com/Whonix/control-port-filter-python/blob/master/usr/share/tor-controlport-filter/examples/40_zeronet.yml

Given your error:

TorManager Tor controller connect error: AttributeError: 'NoneType' object has no attribute 'group' in TorManager.py line 160

which triggers in this part of ZeroNet's src/Tor/TorManager.py:

[...]
# Version 0.2.7.5 required because ADD_ONION support
res_version = self.send("GETINFO version", conn)
version = re.search('version=([0-9\.]+)', res_version).group(1)
assert float(version.replace(".", "0", 2)) >= 207.5, "Tor version >=0.2.7.5 required, found: %s" % version

self.status = u"Connected (%s)" % res_auth
self.conn = conn
except Exception, err:
self.conn = None
self.status = u"Error (%s)" % err
self.log.error("Tor controller connect error: %s" % Debug.formatException(err))
[...]

it seems to me like your filter lacks a rule allowing the "GETINFO version" command.

Cheers!

[Patrick Schleizer]

Post by anonym

Post by Patrick Schleizer
Noticed one incompatibility.
ZeroNet uses custom code rather than python-stem to talk to Tor control
protocol. It's line handling works with original Tor, but not with the
filter.

The filter *should* be able to deal with any client implementation as long as it follow the control-spec, but there can of course be bugs.

Post by Patrick Schleizer
https://github.com/HelloZeroNet/ZeroNet/issues/756
https://github.com/Whonix/control-port-filter-python/blob/master/usr/share/tor-controlport-filter/examples/40_zeronet.yml

TorManager Tor controller connect error: AttributeError: 'NoneType' object has no attribute 'group' in TorManager.py line 160
[...]
# Version 0.2.7.5 required because ADD_ONION support
res_version = self.send("GETINFO version", conn)
version = re.search('version=([0-9\.]+)', res_version).group(1)
assert float(version.replace(".", "0", 2)) >= 207.5, "Tor version >=0.2.7.5 required, found: %s" % version
self.status = u"Connected (%s)" % res_auth
self.conn = conn
self.conn = None
self.status = u"Error (%s)" % err
self.log.error("Tor controller connect error: %s" % Debug.formatException(err))
[...]
it seems to me like your filter lacks a rule allowing the "GETINFO version" command.

I don't think so. It's already white listed here:

https://github.com/Whonix/control-port-filter-python/blob/master/etc/tor-controlport-filter.d/30_whonix.yml

Also no rejected messages in journal. Actually, the communication in the
logs looked all correct.

Best regards,
Patrick

[anonym]

Post by anonym

Post by Patrick Schleizer
https://github.com/HelloZeroNet/ZeroNet/issues/756
https://github.com/Whonix/control-port-filter-python/blob/master/usr/share/tor-controlport-filter/examples/40_zeronet.yml

TorManager Tor controller connect error: AttributeError: 'NoneType' object has no attribute 'group' in TorManager.py line 160
[...]
it seems to me like your filter lacks a rule allowing the "GETINFO version" command.

Actually, this is wrong, cause I now see that that part of the code has changed recently. Rather, I think ZeroNet doesn't follow the control-spec. I pushed a PR to: https://github.com/HelloZeroNet/ZeroNet/issues/756

Cheers!

[Patrick Schleizer]

Noticed one incompatibility.>>
https://github.com/HelloZeroNet/ZeroNet/issues/756

https://github.com/Whonix/control-port-filter-python/blob/master/usr/share/tor-controlport-filter/examples/40_zeronet.yml

anonym sorted that out by fixing a bug in ZeroNet.

https://github.com/HelloZeroNet/ZeroNet/issues/756#issuecomment-274029807

Thanks,
Patrick