error-message
success-message
saving-message
warning-message
JavaScript must be enabled.
There was an error retrieving this content.
The content could not be found.
This is a snapshot of this object as it looked in the past. | Go to the most recent version
WARC writing MITM HTTP/S proxy
Warcprox - WARC writing MITM HTTP/S proxy
.. image:: https://travis-ci.org/internetarchive/warcprox.svg?branch=master :target: https://travis-ci.org/internetarchive/warcprox
Warcprox is an HTTP proxy designed for web archiving applications. When used in
parallel with brozzler <https://github.com/internetarchive/brozzler>
_ it
supports a comprehensive, modern, and distributed archival web capture system.
Warcprox stores its traffic to disk in the Web ARChive (WARC) file format
<https://iipc.github.io/warc-specifications/specifications/warc-format/warc-1.1/>
,
which may then be accessed with web archival replay software like OpenWayback
<https://github.com/iipc/openwayback>
and pywb
<https://github.com/webrecorder/pywb>
. It captures encrypted HTTPS traffic by
using the "man-in-the-middle" technique (see the Man-in-the-middle
section
for more info).
Warcprox was originally based on pymiproxy
<https://github.com/allfro/pymiproxy>
_ by Nadeem Douba.
.. contents::
Warcprox runs on python 3.4+.
To install latest release run::
# apt-get install libffi-dev libssl-dev
pip install warcprox
You can also install the latest bleeding edge code::
pip install git+https://github.com/internetarchive/warcprox.git
To start warcprox run::
warcprox
Try warcprox --help
for documentation on command line options.
Normally, HTTP proxies can't read encrypted HTTPS traffic. The browser uses the
HTTP CONNECT
method to establish a tunnel through the proxy, and the proxy
merely routes raw bytes between the client and server. Since the bytes are
encrypted, the proxy can't make sense of the information that it proxies. This
nonsensical encrypted data is not typically useful for web archiving purposes.
In order to capture HTTPS traffic, warcprox acts as a "man-in-the-middle"
(MITM). When it receives a CONNECT
directive from a client, it generates a
public key certificate for the requested site, presents to the client, and
proceeds to establish an encrypted connection with the client. It then makes a
separate, normal HTTPS connection to the remote site. It decrypts, archives,
and re-encrypts traffic in both directions.
Configuring a warcprox instance as a browser’s HTTP proxy will result in security certificate warnings because none of the certificates will be signed by trusted authorities. However, there is nothing malicious about warcprox functions. To use warcprox effectively, the client needs to disable certificate verification or add the CA certificate generated by warcprox as a trusted authority. When using the latter, remember to undo this change when finished using warcprox.
The warcprox API may be used to retrieve information from and interact with a running warcprox instance, including:
/status
URLWARCPROX_WRITE_RECORD
HTTP methodWarcprox-Meta
HTTP headerFor warcprox API documentation, see: <api.rst>
_.
Warcprox avoids archiving redundant content by "deduplicating" it. The process
for deduplication works similarly to deduplication by Heritrix
<https://github.com/internetarchive/heritrix3>
_ and other web archiving tools:
sqlite <https://sqlite.org/>
_ by default, rethinkdb
<https://github.com/rethinkdb/rethinkdb>
_ with two different schemas, and
trough <https://github.com/internetarchive/trough>
_)revisit
record referencing the url and capture time
of the previous capturea. Write response
record with full payload
b. Store new entry in deduplication database (can be disabled, see
Warcprox-Meta HTTP request header <api.rst#warcprox-meta-http-request-header>
_)
The deduplication database is partitioned into different "buckets". URLs are
deduplicated only against other captures in the same bucket. If specified, the
dedup-buckets
field of the Warcprox-Meta HTTP request header
<api.rst#warcprox-meta-http-request-header>
_ determines the bucket(s). Otherwise,
the default bucket is used.
Deduplication can be disabled entirely by starting warcprox with the argument
--dedup-db-file=/dev/null
.
Warcprox stores some crawl statistics to sqlite or rethinkdb. These are
consulted for enforcing limits
and soft-limits
(see Warcprox-Meta
fields <api.rst#warcprox-meta-fields>
_), and can also be consulted by other
processes outside of warcprox, such as for crawl job reporting.
Statistics are grouped by "bucket". Every capture is counted as part of the
__all__
bucket. Other buckets can be specified in the Warcprox-Meta
request header. The fallback bucket in case none is specified is called
__unspecified__
.
Within each bucket are three sub-buckets:
new
- tallies captures for which a complete record (usually a
response
record) was written to a WARC filerevisit
- tallies captures for which a revisit
record was written to
a WARC filetotal
- includes all URLs processed, even those not written to a WARC
file, and so may be greater than the sum of new and revisit recordsWithin each of these sub-buckets, warcprox generates two kinds of statistics:
urls
- simple count of URLswire_bytes
- sum of bytes received over the wire from the remote server
for each URL, including HTTP headersFor historical reasons, the default sqlite store keeps statistics as JSON blobs::
sqlite> select * from buckets_of_stats;
bucket stats
--------------- ---------------------------------------------------------------------------------------------
__unspecified__ {"bucket":"__unspecified__","total":{"urls":37,"wire_bytes":1502781},"new":{"urls":15,"wire_bytes":1179906},"revisit":{"urls":22,"wire_bytes":322875}}
__all__ {"bucket":"__all__","total":{"urls":37,"wire_bytes":1502781},"new":{"urls":15,"wire_bytes":1179906},"revisit":{"urls":22,"wire_bytes":322875}}
Warcprox supports a limited notion of plugins by way of the --plugin
command line argument. Plugin classes are loaded from the regular python module
search path. They are instantiated with one argument that contains the values
of all command line arguments, warcprox.Options
. Legacy plugins with
constructors that take no arguments are also supported. Plugins should either
have a method notify(self, recorded_url, records)
or should subclass
warcprox.BasePostfetchProcessor
. More than one plugin can be configured by
specifying --plugin
multiples times.
See a minimal example here
<https://github.com/internetarchive/warcprox/blob/318405e795ac0ab8760988a1a482cf0a17697148/warcprox/__init__.py#L165>
__.
.. image:: arch.svg
Warcprox is multithreaded. It has pool of http proxy threads (100 by default). When handling a request, a proxy thread records data from the remote server to an in-memory buffer that spills over to disk if necessary (after 512k by default), while it streams the data to the proxy client. Once the HTTP transaction is complete, it puts the recorded URL in a thread-safe queue, to be picked up by the first processor in the postfetch chain.
The postfetch chain normally includes processors for loading deduplication
information, writing records to the WARC, saving deduplication information, and
updating statistics. The exact set of processors in the chain depends on
command line arguments; for example, plugins specified with --plugin
are
processors in the postfetch chain. Each postfetch processor has its own thread
or threads. Thus the processors are able to run in parallel, independent of one
another. This design also enables them to process URLs in batch. For example,
the statistics processor gathers statistics for up to 10 seconds or 500 URLs,
whichever comes first, then updates the statistics database with just a few
queries.
Warcprox is a derivative work of pymiproxy, which is GPL. Thus warcprox is also GPL.
This program is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 2 of the License, or (at your option) any later version.
This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.
You should have received a copy of the GNU General Public License along with this program; if not, write to the Free Software Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
This section shows a list of objects the current object is dependent upon in order to be used.
For dependencies that are only relevant as part of the build or runtime of the object, it is best to describe those local to those sections.
language
python
3.7.x
python-library
certauth
==1.1.6 || ==1.1.6.0
python-library
warctools
>=4.10
python-library
urlcanon
>=0.3
python-library
doublethink
>=0.2.0.dev87
python-library
urllib3
>=1.14
python-library
requests
>=2.0.1
python-library
PySocks
>=1.6.8
python-library
cryptography
>=2.3
python-library
idna
>=2.5
python-library
PyYAML
>=5.1
python-library
cachetools
>=0
This section tracks information useful to describing how to build this object.
warcprox-2.4.17.tar.gz
application/gzip
Python Source for warcprox 2.4.17
https://files.pythonhosted.org/packages/97/2a/699279920300e3e100ec017162a0cf0fe5c6a8b1fbf0328d282459061892/warcprox-2.4.17.tar.gz
QmRKKGrd9ZYjYo2umUED3FF2MaN4cyTrkmWEDktgNbejtp
QmNaDDxhgRA4VBUbeGUzy3W4kMYNJW6uHmr4d4oz22eijz
QmXoUyjnrxDzvq5zz3f8brvCbsgyjR2vcRKx3KzsEeGETo
downloadargparse-1.4.0.tar.gz
application/gzip
Python Source for argparse 1.4.0
https://files.pythonhosted.org/packages/18/dd/e617cfc3f6210ae183374cd9f6a26b20514bbb5a792af97949c5aacddf0f/argparse-1.4.0.tar.gz
QmVMjtpCHUfoKdHkNdqDCry1Ck6P8Tk9bJaHmrH7a35oXr
QmYhLtiLR6hamZiy5TL3vf5d6fA1z2eq4iGn4EmeJywkFs
QmUyto3kPC3XY43kf9bh6VoSJnEVvUCX32XX5EPXZCFCNw
download/bin/bash
{{ paths.mount }}/build.sh
warcprox-2.4.17/
3
warcprox
argparse-1.4.0.tar.gz
linux
x86-64
Objects have a set of files that comprise that object, as seen in the Files tab. When an object is built, they have a set of files that are the result of that build. In each case, these sets of files are the ones that are accessible within a virtual machine when running the object.
This section describes how those files are accessible and visible to that machine. When a virtual machine is created, the data is always accessible read-only within a particular mounted directory. However, applications may expect to be located at particular directories. This section allows one to place files in different directories when the virtual machine is launched.
There are two types of access. One is a link where the file is not copied from its original location. Instead, it creates what is called a symbolic link which is a "shortcut" that acts like a normal file but allows the actual data to be in a different directory. The other type is a traditional copy which copies the data explicitly to the provided destination. This takes more time, but the copy can be modified, unlike the read-only link type.
Action | Source | Destination | Summary | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
Link | usr | /usr | N/A | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
No files will be installed. |
JavaScript must be enabled.
There was an error retrieving this content.
The content could not be found.
JavaScript must be enabled.
There was an error retrieving this content.
The content could not be found.
JavaScript must be enabled.
There was an error retrieving this content.
The content could not be found.
Confirm message?