Discussion:
[htp] 0001-When-date-header-cannot-be-fetched-print-URL-too.patch
(too old to reply)
Linus Nordberg
2018-03-17 16:15:10 UTC
Permalink
Raw Message
Hi devs,

Please find attached a very minor patch for slightly improved
diagnostics in htp.

Since late February, I've seen 75 requests from 18 sites fail to
retrieve a date header:

--8<---------------cut here---------------start------------->8---
cat /var/log/htpdate.log | awk '/Could not get any Date header from/{print $8}'| sort | uniq -c | sort -rn
8 https://encrypted.google.com
6 https://www.stackexchange.com
6 https://www.paypal.com
6 https://www.myspace.com
6 https://twitter.com
5 https://www.duckduckgo.com
4 https://www.sony.com
4 https://tumblr.com
4 https://login.yahoo.com
4 https://lkml.org
4 https://espiv.net
3 https://www.privacyinternational.org
3 https://www.mozilla.org
3 https://www.eff.org
3 https://www.1984.is
2 https://www.immerda.ch
2 https://thepiratebay.org
2 https://en.wikipedia.org
--8<---------------cut here---------------end--------------->8---

I'm afraid I don't have any numbers on how often date header fetches
From the above sites _succeed_ so maybe not so useful.
intrigeri
2018-03-21 15:11:30 UTC
Permalink
Raw Message
Hi Linus,
Post by Linus Nordberg
Please find attached a very minor patch for slightly improved
diagnostics in htp.
Thanks! Applied to htp.git and then imported the latest version of
the script into tails.git (devel branch).
Post by Linus Nordberg
Since late February, I've seen 75 requests from 18 sites fail to
I suspect some of these failures are caused by a too strict regexp:
some servers send a "date:" header while the code you're running only
looks for a "Date:" header. We've fixed that problem in tails.git
7 months ago but back then we failed to merge the corresponding change
into htp.git; apparently due to a (lack of) communication problem
between the developer and the reviewer. So I've just done this merge.

I'm curious about how your results will look like once you upgrade to
what's on the master branch in htp.git now :)

Cheers,
--
intrigeri
Loading...