Note: This post describes automated statistics we used to interpret scores of reviews to guide us in assigning more reviews or guiding discussion. All final acceptance decision were taken the old fashioned way through qualitative assesment of the reviews and discussion in the PC.

A naïve reviewing strategy would have been straight forward: traditionally each paper has to receive 3 reviews, leading to a load of 23-24 reviews per PC member. This is a high, but not impossible load. Yet it would not have led to the best outcome for a number of reasons:

• Papers with conflicting reviews are best discussed by requesting additional reviews. This leads to a large fraction of papers needing in fact 4 or more reviews, stretching further the pressure on the members of the PC.
• Author rebuttals to reviews require careful consideration from independent members of the PC, to judge the validity of the author’s response over the previous reviews and paper. This further increases the need for reviews on some papers – authors need at least a couple of reviews to respond to, and a further round of reviewing increases the load for papers with conflicting reviews.
• Finally, it would be nice to assign most papers that have a good chance of being accepted more reviews than the baseline three. This is to make sure that papers are of high quality: identify subtle overlaps with existing work to be mentioned, and provide authors of viable papers with as much feedback as possible to improve their work in case it was accepted.

Given the above, even if only 200 papers needed 4 reviews, we would be asking our PC to review about 27 papers each. This would be getting too heavy, and likely to impact the quality of all reviews. In practice we knew that many papers would need even more reviews. Since the straightforward naïve strategy was not possible, here is a description of the strategy followed, and my personal rationale behind it. Hopefully this will lead to discussion on its effectiveness, fairness and possible improvements for future years.

Broadly speaking the strategy has been to concentrate reviewing resources and discussion at any time where needed to decide whether the paper should be accepted or not. This was done in an adaptive manner given the information available at any time.

First of all the decision was taken to give some papers only 2 or even one reviews in the first phase. This was a time consuming manual task: PC chairs looked at all submissions and flagged some to receive fewer reviews. Factors taken into account were possible lack of an evaluation section, no mention of security or privacy, no references to the security / cryptography literature, no security argument, number of members of the PC willing to review the paper etc. These issues were all considered as a whole when making this decision. Over 100 papers were manually flagged for a reduced number of reviews initially.

In the first round of reviews each PC member received about 20 reviews. We also allocated a large number of reviews to external reviewers, many of whom accepted to do 3 or 4 reviews. Unlike previews conferences I have managed, no attempt was made to keep the load overly fair in terms of numbers of papers: some reviewers got as few as 15 papers while others did over 25 reviews in the first phase. The rational being that it is less onerous, and provides best results, to review papers on one’s field than even a handful of papers in an unknown field.

By the time reviews were sent to authors most papers had already received at least 2 reviews – sadly many late reviews were not given to authors to avoid delaying the rebuttal deadlines. Once rebuttals came in, the second phase of reviewing started, involving further reviews and active discussion.

Controversially, we asked the second phase reviewers to have a look at the previous reviews and author rebuttals. Their job was not only to assess the paper, but also to discuss with previous reviewers, and the answers authors had provided in their rebuttal. At the same time, this invalidates the usual assumption that reviews are independent. In practice reviews and scores get changed so often during the discussion phase (and this is a good thing!) that this is unlikely to severely distort the results.

This is where things get tricky: after rebuttals we had a collection of 429 papers, some with 1, some with 2, many with 3 and some with more reviews. How do we know which papers need more reviews? How do we know if a paper with one bad review may simply have been unlucky? How do we know whether a paper with a certain number of reviews is likely to make it in the top 60 that are to be accepted, particularly when the reviewers’ confidences are different?

Of course, reading the reviews gives a lot of qualitative information to answer the above questions. Some reviews and comments explicitly were requesting more reviews. At all stages of the process any PC member could simply request more reviews for any paper. Qualitative factors and the view of the PC at the end decided which papers needed more reviews as well as the final outcome for a paper.

At the same time, it is genuinely hard to judge where a paper (out or many hundred) sits in comparison with the others given a set of often contradictory scores. It is also genuinely hard to get intuitions about how likely it is that a paper has received a few bad (or good) reviews by chance or by mistake, i.e. the natural variance of a small set of reviews. Taking into account reviewer confidence when judging paper ranking is difficult and poorly supported in conference management systems.

To help with this task, along with the quantitative feedback, we devised a system to estimate the ranking of papers – for the purpose of assigning more reviews (not acceptance!). Broadly, data sets of reviewer scores and confidences were used from this year’s CCS as well as past years to build a model of the range of scores we expect each paper to be taking if more reviews were to be assigned. Then we used a Monte-Carlo approach to get an estimate of the rank of the paper as well as confidence intervals on this estimate (and any other desirable statistic).

A few more words on the model used, as well as its rationale:

• We consider each paper in turn and sample sets of scores it would have, given the scores so far, up to 5 scores.
• To do this we first apply a stage a bootstrapping: we sample from the list of scores it has with replacement the same number of scores. Scores include the confidence of the reviewer (i.e each score is a tuple (score, confidence)). This stage is meant to smooth out the influence of any individual a-typical review.
• Then we take the re-sampled scores, and assign new scores until we have 5 scores. The assignment is done by taking two random existing scores (and their confidences), and conditioning the new score on these scores. The dataset of the 3 last CSS conferences was used to achieve this.
• Rejection sampling was used to ensure we do not end up with a-typical triplets (i.e. we have seen a tuple of scores before on real data a few times).
• This procedure was repeated 2000 times for each paper, creating 2000 different “scenarios” of rankings for all papers.
• For each paper we then count how often it would have been in the top 60 papers (per score only). That gives us a measure of the probability of being included in the final program.
• By taking the ranking of a paper in all 2000 scenarios we can calculate the confidence intervals (CI) of the paper rank.

All the above modelling steps are conservative to increase the variance in the distribution of scores of a paper.

A set of heuristics was then used to determine which papers needed more reviews. Any papers with probability lower than 1% of being included in the program suffered a sudden death: unless the reviews were ambivalent it received no further reviews. Any papers with probability between 1% and 10% received at least 3 reviews, and any papers with probability over 10% received more reviews. Of course these were only heuristics – the qualitative information in the reviews, and the express feedback from reviewers guided heavily the assignments of additional reviews.

It is informative to give a feel for the distribution of papers according to their respective probabilities of acceptance using the above model. At the very end of the reviewing process, about 50 papers had probability more than 50% of being accepted. About 125 papers had probability higher than 10% of being accepted and 194 papers had a probability greater than 1%. The last number is less than half of the submitted papers.

This is mildly encouraging: it means that the reviewing process actually does yield value – the posterior probability distributions of papers (taking into account the variance and conditioning on scores) is more informative than the prior probability of 60 / 429, even if we take into account the variance of reviews.

When one compares the probabilities given by the model with the actual acceptances and rejections of papers, it is clear that there has been a slight bias in accepting papers with higher probabilities of acceptance (We would have expected to choose about 43 papers from the 60 papers with highest acceptance probability, but in fact 48 were selected). This bias may be due to the qualitative discussion around those papers, or it might be due to reviewers paying too much attention to the actual scores and specific reviews – without contextualizing them in relations to their natural variance.

Since the model gives us a probability of acceptance for each paper, we can use it to estimate the probability of error in the program. Assuming the model is correct, we have made an expected 20 out of 60 mistakes in our selection of papers in the worst case! Reality of course is less tragic, as the model takes no account of the qualitative feedback and discussion, but the potential magnitude of the error is a sobering realization that scores alone are a poor way to choose papers. (Note that if we were selecting papers at random we would have made 52 mistakes on average – so it’s still much better than random). Given just the score information, we would have to accept at least 120 papers to ensure that 55 of the top 60 papers were included in the program on average – an enormous cost.

The above really illustrates the importance of deep engagement in the discussion phase. After the second round of reviews came back, most papers were discussed in some way. Discussions went deep into the technical contributions of the paper, and reasons for rejecting papers had to be documented and cross checked by all reviewers. In total 1972 comments were made on papers and the number of comments is correlated with the score as well as the variance of reviews. Given the natural variance of mere scores, this makes all the difference.

It is also interesting to estimate the cost of the process on the community of volunteers. If we assume that each review takes at least 3 hours, and that each comments takes at least 5 minutes, the review process has taken about 1.8 person work years (300 days / 8 hours a day). If we cost the opportunity cost of each reviewer at $100K a year, the process has cost $177K or a minimum of $413 per submission. This volunteer commitment is worth keeping in mind when debating the cost of academic publishing, as well as conferences.

In conclusions: the rebuttal phase followed by further reviews has been a positive experiment, and requiring 2^nd round reviewers to comment on previous reviews and rebuttals ensures that the author opinions are taken seriously. The decision to focus reviewing effort where uncertainty was, is controversial, and was supported by both time-consuming qualitative understanding of the papers and the reviews, as well as quite advanced quantitative models for the natural variance of papers given a set of scores. The latter can only be used as a guide to commit reviewing effort. Scores on their own do not give a very good indication of which papers should be included in the program and which should be excluded. Concentrating eyeballs where the stronger papers were was overall a very good idea, and a robust and lengthy discussion amongst the PC is absolutely essential. It is still astonishing that more than half the papers submitted had less than 1% probability of finding their way into the program (despite the variance of reviews). Even more astonishing is the potential expected error from the selection process – one more reminder that acceptance (or rejection) from a specific venue, no matter how competitive, is only one hint about the ultimate quality of the work.

Finally as a reward for reading all the way through the post here is some data for the geeks! That can hopefully illuminate the debates about reviewing with some facts:

1. A table illustrating the natural variance of scores in the past few CCS conferences.

   A simplified table from the CCS11 reviewing process (and previous CCS)
   showing the distribution of other scores (columns), given a (score, confidence)
   pair in a review (rows). Entries with less than 4 have been redacted to
   "x" to illustrate they are not significant at all and preserve privacy.
   
   Note: The actual model used to estimate missing scores from CCS11 submissions
   was taking into account 2 pairs of (score, confidence) at a time and
   mapping them to another (score, confidence) pair with the appropriate
   empirical probability. This full table is not shown here.
   
           -3   -2   -1    0    1    2    3
   -3(1)    7   10    4    x    x    x    x
   -3(2)   22   31    x    x   10    x    x
   -3(3)   87  147   34   19    9    x    x
   -3(4)   84  115   34   11   15    9    x
   -2(0)    x    x    x    x    5    x    x
   -2(1)   17   45   26    x   18    4    x
   -2(2)   50  207  185   68   41   32    x
   -2(3)  154  454  419  161  150   70    4
   -2(4)   82  225  180   75   72   40    4
   -1(0)    x    x    x    5    x    x    x
   -1(1)   12   48   50   12   15    5    x
   -1(2)   24  310  167   95  128   52    7
   -1(3)   36  350  238  190  197  102    6
   -1(4)    x  102   62   65   48   26    x
   0(1)    7   29   23   18   17   11    x
   0(2)   18  137  150   88   85   63    5
   0(3)    x  117  136   91   99   85    5
   0(4)    4   22   58   32   26   15    x
   1(0)    x    4    x    x    4    x    x
   1(1)    9   38   29   14   23   29    x
   1(2)   16  115  159   84   97   76   12
   1(3)   11   95  154  102  113  105    9
   1(4)    x   34   47   27   35   35    x
   2(0)    x    x    x    x    x    x    x
   2(1)    x   17   22   10   12   20    x
   2(2)    7   59   85   56   83   68   17
   2(3)    4   60   60   81  104   81   15
   2(4)    4   11   17   27   46   40    6
   3(1)    x    x    x    x    x    x    x
   3(2)    x    7   12    x    x   11    4
   3(3)    x    x    x    4   14   22    7
   3(4)    x    x    x    4    6    6    x

2. A graph of the ordering of CCS11 papers by rank according to the model, and the 90% CI of the rank. Acceptance / Rejection decision is not given — this information was used to inform assigning reviews, not acceptance.
   
3. A table showing the number of papers within 10% bands of acceptance probabilities according to the model, as well as the number accepted in each band.

   The distribution of accept probabilities for CCS11 submissions given the score model used to assign further reviews.
   
   Columns:
   Accept prob.: The probability of the paper being in the top-60 papers by score, given the model of scores.
   Sub.: The number of submissions with a probability in this range.
   Accepts: The number of accepted papers with a probability in this range.
   
   Accept prob.	Sub.	Accepts
    0% -  9%:	302	1
   10% - 19%:	40	2
   20% - 29%:	16	2
   30% - 39%:	17	9
   40% - 49%:	6	3
   50% - 59%:	10	8
   60% - 69%:	8	6
   70% - 79%:	7	7
   80% - 89%:	6	6
   90% - 99%:	14	13
   100%:		3	3

4. A table showing a histogram of the number of comments, the number of papers for a certain number of comments, the average score of the papers, the average difference between the high and low review and the average number of reviews.

   Histogram of number of comments during the ACM CCS11 review process
   
   Columns:
   - Comments: number of comments
   - Papers: number of papers with a certain number of comments
   - Av.Sc.: The average score of the papers in this bucket.
   - Av.Dif.: The average difference between high and low score of the papers in this bucket.
   - A.Revs: The average number of reviews of the papers in this bucket.
   
   Comments	Papers	Av.Sc.	Av.Dif.	Av.Revs
   0		112	-1.98	0.78	2.3
   1		52	-1.49	1.35	2.8
   2		44	-1.08	1.48	3.0
   3		42	-0.62	2.38	3.3
   4		30	-0.24	2.03	3.6
   5		23	-0.21	2.00	3.7
   6		16	-0.40	2.38	3.8
   7		18	0.18	2.61	3.8
   8		17	0.28	2.29	3.9
   9		13	0.08	2.85	3.8
   10		7	0.22	2.57	4.1
   11		15	0.06	3.07	3.9
   12		5	0.65	2.60	4.0
   13		5	0.65	2.40	4.0
   14		3	XXXX	XXXX	4.0
   15		7	0.24	2.00	3.7
   16		20	0.62	3.00	4.5

Privacy in for smart electricity provision seems to be a rising topic, and this year there is a whole session on it at PETS 2011. The first paper (one which I am a coauthor) looks at the problem of gathering aggregate data from groups of smart meters, without allowing any third party to get the the individual measurements. This can be applied as a PET to solve real-world problems such as fraud detection, leakage detection, load estimates, demand response, weather prediction — all of which only require aggregate data (sometimes in real time), not individual measurements.

The key challenge to providing a private aggregation protocols are the specific constraints of smart meters. They are cheap devices, with modest resources, hardly any bandwidth, no ability to communicate, etc. Two specific protocols are presented: the first one allows to compare the sum of meter readings with a reference number (maybe measured from a feeder meter). This protocol allows for fancy proofs of correctness, but it slow in terms of computation and bandwidth (it requires public key operations for each reading). The second protocol is extremely fast and has no communication overhead. In both cases a pragmatic approach to the threat model is followed: we assume that the utilities will be honestly defining groups of meters and facilitating the key management protocol — for the second protocol there is no overhead of public key operations after the initial key setup.

The key highlight from this work is not as much its technical depth (tricks with DC networks and hash function that would not surprise any PETS regular). What is interesting is that the protocols were designed for a real industrial application and now fully integrated on real smart meters and their communication protocols in collaboration with our collaborators at Elster.

Plug-in privacy for Smart Metering billing
Marek Jawurek, Martin Johns, and Florian Kerschbaum (SAP Research)

This second paper looks at the problem of billing for fine-grained time of use tariffs — their energy consumption at different times costs a different rate per unit. This is a very important topic, as correct billing and time of use tariffs are a key driver of fine-grained data collection through smart meters — if we can do billing privately then maybe less personal information may be collected.

Technically the protocols proposed are based on the homomorphic properties of Pedersen commitments: readings are commitments, and you can use multiplication by a constant and addition to compute the bill, and (most importantly) prove that it is correct. The system model is that the meter outputs signed commitments of readings, a privacy component computes the bill and proofs of correctness, and those are sent to the supplier for verification (and printing the bills!).

This is the core of a nice solution for the basic billing case (which is likely to be the common one in smart grids). We have shown in related work that the protocol can be further improved to have zero communication overhead. Since it avoids expensive zero-knowledge proofs it is fast for its proofs and verification. It also provides the basic infrastructure to support further more expressive billing policies and general computations.

Traditional entropy based anonymity metrics look at the security of single messages. But how can you quantify the security provided by a whole system? The first paper in this session looks at a system-wide definition of anonymity by “counting” the possible number of matchings between inputs and outputs of an anonymity system. Furthermore, the metric extends to the probabilities over perfect matchings to express subtleties of modern anonymity systems. The paper first of all provides a thorough critique of the metric by Edman et al. (there was also previous work on this metric by the Leuven crew).

In a nutshell the proposed system-wide metric associates a probability to each possible matching, and computes the entropy over this distribution as a measure of anonymity (normalized). The choice of shanon entropy to summarise quality can be changed to min-entropy or other (which is very cool!) One key issue with system-wide metrics is that  how they express the properties that any individual message receives. Paul Syverson points out that these type of metrics express more the anonymity capacity of a system — namely how much anonymity the system could provide as a whole. The question of how this capacity for protection is distributed across users may need an extension to those metrics. For anyone who would like to extend metrics to capture this aspect, this paper is a very solid foundation.

DefenestraTor: Throwing out Windows in Tor
Mashael AlSabah, Kevin Bauer and Ian Goldberg (University of Waterloo), Dirk Grunwald (University of Colorado), and Damon McCoy, Stefan Savage, and Geoffrey Voelker (University of California-San Diego)

This paper looks at performance issues within the Tor network, and in particular the effects of the congestion and flow control protocols. Tor implements simple end-to-end flow control mechanism at the granularity of circuits and streams. It turns out that the implemented window based flow control has detrimental effects on performance: it does not protect intermediate routers (who are likely to be the congested ones) from congestion.

Two approaches were followed to solve this problem. First, a smaller window could be used — but this would not solve the problem; or windows can be computed dynamically. Second, the N23 congestion control protocol (used for ATM) could be used over Tor. N23 is simple and guarantees no packets are dropped, while implementing a steady flow of data. Its a credit based system, where packets are sent when credits are available (and consume them), and credits are sent up the network when bandwidth is available.

The evaluation was done under realistic conditions on ExperimenTor. The improvement over the current Tor strategy is significant when it comes to the time to get the first byte, but the time to complete larger (bulk) downloads do suffer (which is part of the point of the protocol).

I am really happy to see research on the intersection of traditional networking and anonymous communications. I have never heard of N23 before (shame on me!), and it seems that it is a good fit for the problem of congestion in anonymity networks (where reliability is not an issue when TCP is used).

Privacy Implications of Performance-Based Peer Selection by Onion Routers: A Real-World Case Study using I2P
Michael Herrmann and Christian Grothoff (Technische Universität München)

This is an attack paper on the I2P network, and in particular the performance based peer selection. It combines a denial-of-service attack to influence the selection of peers within the network, and force a victim to choose corrupt servers.

This is a cute attack that combines denial-of-service, traffic analysis for confirmation you are on the same circuit, and interactions with an infrastructure to attack. This is a very good reminder that anonymity engineering is not simply systems’ work. Every design choice about performance can affect security in dramatic ways. The evaluation was also very sensitive to protecting users: the researchers tried their attack on the real network, but targeted their own circuits (I still want to see details to make sure no other users were affected).

Tor too implements circuit selection on the basis of performance — I am wondering to what extent similar ideas could be applied there …

This work evaluates the privacy of using location-based services sporadically using a set of location privacy mechanisms. Sporadic services include those that require location infrequently, rather than continuously (think of restaurant suggestions rather than relaying real-time GPS streams). The key novelty of the approach is that the model of location exposure, as well as privacy protection is very general. It encompasses anonymization, generalization and obfuscation of location, use of fake traffic and suppression of location. In turn the analysis relies on advanced models of location and mobility (based on markov chains) and is based on Bayesian inference. The evaluation of different location privacy techniques is done on real-world traces from SF taxis.

I am one of the authors of this work, so of course I think it is awesome! More seriously, it is one of the fist works to combine under a common framework a multitude of location privacy mechanisms, and provide a common evaluation framework for them, to quantify the degree of protection they offer relatively to each other for different adversaries. It is also one of the first systematic applications of Bayesian inference to analyze location privacy — extending the inference paradigm beyond the analysis of network anonymity systems.

Of course this is not the last word. Only a subset of protection techniques and combination of techniques were look at, and other protection mechanisms can be integrated and evaluated in the same framework (the tracing model and threat model can be unchanged). Secondly, the analysis itself may be augmented with side-information — be it commercial transactions or traces of network traffic — that may be giving some information about location, to increase the capabilities of the adversary (or make them more realistic). The model we use, based on markov chains, has the benefit of giving analytically tractable results, but numerical techniques may be used to extend it to be more true to real-life attacks.

The Location-privacy Meter tool that can be used to evaluate custom location privacy protections is available for download!

Privacy in Mobile Computing for Location-Sharing-Based Services
Igor Bilogrevic and Murtuza Jadliwala (EPFL), Kubra Kalkan (Sabanci University), Jean-Pierre Hubaux (EPFL), and Imad Aad (Nokia)

This paper looks at applications where users need to share their location. For example, two users may want to find out if they are close to each other or where they should meet in order to share a taxi ride. Yet, those users do not want to leak any of their location information to the other users or the service provider. More specifically two users specify a set of ranked prefered location they could meet and the system needs to determine on of those fairly without revealing the current location or other preferences (except the one chosen to meet). This is called the fair rendez-vous problem.

The key contribution of this work is to show that this problem can be set with a set of concrete cryptographic protocols. It also presents an implementation of these algorithms on a real mobile phone to show that it is practical. The cryptographic computations are based on homomorphic encryption schemes as well as interactions with the service (to do multiplication that is not possible with Paillier). The implementation on a mobile phone takes a few seconds on the client and the server, and is paralelizable in the number of users. Untypically, the authors also did a user study: users were asked what their concerns were, and after using the application of the phone they were asked how usable it was, and whether they appriciated the privacy provided by the application.

This is a really nice example of a privacy application, that applies advanced crypto, but also evaluates it on a real platform for performance as well as users’ reaction to it. The obvious extensions to this work would be to generalize it to more complex rendez-vous protocols, as well as other location sharing applications. It is good to see that modern mobile devices can do plenty of crypto in a few seconds, so I am very hopeful we will see more work in this field.

On The Practicality of UHF RFID Fingerprinting: How Real is the RFID Tracking Problem?
Davide Zanetti, Pascal Sachs, and Srdjan Capkun (ETH Zurich)

This paper looks that UHF tags — they are the dumb tags that can be read at about 2m that are attached to things you buy to facilitate stock management or customer aftercare. Interestingly this study looks at how identifiable the tags are at the physical layer, not using the actual tag ID! Therefore these techniques may bypass any privacy protection that attempt to prevent access to the tag ID. It turns our that one can build a unique and reliable ID for a tag from its physical characteristics that can be used to trace people as they move around.

What is new about this work is that the focus was on practicality and cost of extracting a reliable fingerprint (previous approaches relied on expensive equipment and laboratory conditions). The solution was implemented using a cheap software radio (USRP2 device + PC).

I am not quite sure what to conclude from the evaluation on the quality of the fingerprint. It seems that an adversary can place tags within one of 83 to 100 groups. Is this really a good results or not? I guess it depends on the application and the density of tags. Of course if more than one tag is carried, then the adversary could combine fingerprints to identify individuals more easily — if you carry 5 tags you have a 20 bit IDs. Interestingly, there is extensive evaluation of the stability of the tag to temperature and mobility — it turns out that these factors do affect the quality of the fingerprint and further reduce the effective number of unique IDs that can be extracted (down to about 49 classes).

It would be interesting to combine this attack vector with the ideas from the first paper (pretending that the short physical IDs are a version of a privacy protection system) to evaluate the effectivness of tracing a set of individual throughout town.

The pannel includes a short excerpt from an interview with Andreas, as well as recorded contributions, by collaborators (Michael Waidner and Marit Hansen), former students (Anna Krasnova and Hannes Federrath) and people in the PET community (Paul Syverson and Caspar Bowden).

The first session is about data mining and privacy.

“How Unique and Traceable are Usernames?”
Daniele Perito, Claude Castelluccia, Mohamed Ali Kaafar, and Pere Manils (INRIA)

The first paper looks at the identifiably of on-line usernames. The authors looked at user names from different sites and assess the extent to which they can be linked together, as well as link them to a real person. Interestingly they used Google Profiles as ground truth, since they allow users to provide links to other accounts. First they assess the uniqueness of pseudonyms based on a probabilistic model: a k-th order markov chain is used to compute the probability of each pseudonym, and its information content (i.e. -log_2 P(username)). The authors show that most of the usernames observed have “high entropy” and should therefore be linkable.

The second phase of the analysis looks at usernames from different services, and attempts to link them even given small modifications to the name. The key dataset used was 300K google profiles, that list (sometimes — they used 10K tuples of usernames) other accounts as well. They then show that the Levenshtein distance (i.e. edit distance) of usernames from the same person is small compared to the distance of two random user names. A classifier is built, based on a threshold of the probabilistic Levenshtein distance, to assess whether a pair of usernames belongs to the same or a different user. The results seem good: about 50% of usernames are linkable with no mistakes.

So what are the interesting avenues for future work here? First, the analysis used is a simple threshold on the edit distance metric. It would be surprising if more advanced classifiers could not be applied to the task. I would definitely try to use random forests for the same task. Second, the technique can be used for good not evil: as users try to migrate between services, they need an effective way to find their contacts — maybe the proposed techniques can help with that.

“Text Classification for Data Loss Prevention” (any public PDF?)
Michael Hart (Symantec Research Labs), Pratyusa Manadhata (HP Labs), and Rob Johnson (Stony Brook University)

The paper looks at the automatic classification of documents as sensitive or not. This is to assist “data loss prevention” systems, that raise an alarm when personal data is about to be leaked (i.e. when it is about to be emailed or stored on-line — mostly by mistake). Traditionally DLP try to describe what is confidential through a set of simple rules, that are not expressive enough to describe and find what is confidential — thus the authors present a machine learning approach to automatically classify documents using training data as sensitive or not. As with all ML techniques there is a fear of mistakes: the technique described tries to minimise errors when it comes to classifying company media (ie. public documents) and internet documents, to prevent the system getting on the way of day to day business activities.

The results were rather interesting: the first SVN classifier looked at unigrams with binary weights to classify documents. Yet, it first had a very high rate of false positives for public documents. It seems the classifiers also had a tendency to classify documents as “secret”. A first solution was to supplement the training set with public documents (from wikipedia), to best described “what is public”. Second, the classifier was tweaked to (in a rather mysterious way to me) by “pushing the decision boundary closer to the true negative”. A further step does 3-category classification as secret, public and non-enterprise, rather than just secret and not-secret.

Overall: They manage to get the false positive / false negative rate down to less than 2%-3% on the largest datasets evaluated. That is nice. The downside of the approach, is common to most work that I have seen using SVNs. It requires a lot of manual tweaking, and further it does not really make much sense — it is possible to evaluate how well the technique performs on a test corpus, but difficult to tell why it works, or what makes it good or better than other approaches. As a resut, even early positive resutls should be considered as preliminary until more extensive evaluation is done (more like medicine rather than engineering). I would personally like to see a probabilistic model based classifier on similar features that integrates the two-step classification process into one model, to really understand what is going on — but then I tend to have a Baysian bias.

P3CA: Private Anomaly Detection Across ISP Networks
Shishir Nagaraja (IIIT Delhi) and Virajith Jalaparti, Matthew Caesar, and Nikita Borisov (University of Illinois at Urbana-Champaign)

The final paper in the session looks at privacy preserving intrusion detection to enable cooperation between internet service providers. ISPs would like to pool data from their networks to detect attacks: either because the volume of communications is abnormal at certain times, or because some frequency component is odd. Cooperation between multiple ISPs is important, but this cooperation should not leak volumes of traffic at each IPS since they are a commercial secret.

Technically, privacy of computations is achieved by using two semi-trusted entities, a coordinator and key holder. All ISPs encrypt their traffic under an additive homomorphic scheme (Paillier) under the keyholder key, and send it to the coordinator. The coordinator is using the key-holder as an oracle to perform a PCA computation to output the top-n eighen vectors and values of traffic. The cryptographic techniques are quite but standard, and involve doing additions, subtraction, multiplication, comparison and normalization of matrices privately though a joint private two-party computation.

Surprisingly, the performance of the scheme is quite good! Using a small cluster, can process a few tens of time slots from hundresds of different ISPs in tens of minutes. A further incremental algorithm allows on-line computations of eighen vector/value pairs in seconds — making real-time use of the privacy preserving algorithm possible (5 minutes of updates takes about 10 seconds to process).

This is a surprising result: my intuition would be that the matrix multiplication would make the approach impractically slow. I would be quite interested to compare the implementation and algorithm used here with a general MPC compiler (under the same honest-but-curious model).

Shishir Nagaraja: Anonymity in the Wild: Mixes on Unstructured Networks. Privacy Enhancing Technologies 2007: 254-271 [pdf][ppt]

This is important prior work and we should have cited it properly. It presents an analysis of an anonymity provided by different synthetic social network topologies, as well as real-world data from LiveJournal.

First, what is the problem? Smart-meters are to be fitting in most homes, and they provide facilities for recording fine-grained readings of energy consumption. These are to be used for time of use billing, energy advice, the backend settlement process, financial projections of suppliers, fraud detection, customer service, and network management. The problem is that these readings are also personal data, and leak information about the occupancy of households, devices used, habits, etc. So here we have a classic privacy dilemma: where to strike the balance between the social value of sharing data (even mandating such sharing) versus the intrusion to home life?

Or do we? As it is often the case when privacy is framed as a balance, what is ignored is that we can use technology to achieve both privacy and extract value from the data. In fact we show no balancing act is necessary. We designed a host of privacy technologies to fulfill the needs of the energy industry (even the rather exotic ones) while preserving extremely high levels of privacy and user control. Lets look at them in detail:

• We developed a set of protocols to perform computation on private data while maintaining a high degree of integrity and availability. This allows customers to calculate their bills, provide indicators of consumed energy value to be used in settlement, routing demand response requests, and do profiling to support network operation or even marketing. Our framework guarantees that the computations only leak their results to third parties, and also that those results are in fact derived from the real meter readings. The raw meter readings are not necessarily shared, but can be used locally on any user client to offer a rich experience — i.e. pretty graphs of consumption and comparison with their neighbours. A non technical overview is available as a white paper, a technical introduction for meter manufacturers is provided, and a preliminary technical report with all the crypto is also online.
• Sometimes it is important to aggregate information from multiple meters without revealing anything about individual readings. The traditional approach has been to give all readings to a trusted third-party that performs the aggregation and only publishes the sum. We show that a set of meters can in fact perform the aggregation without the need for a trusted party. This is simple, efficient and compact — the computations can be done inside the trusted meter or outside along with cryptographic verification. All details are available in our technical report on aggregation.
• Some smart-meters may be deployed in extremely high-security settings. In such places leaking even the final bill or statistics aggregated over time may leak information and a positive guarantee that the information leakage is limited might be necessary. We developed techniques inspired from differential privacy to inject noise to aggregate readings that guarantee any specific time period consumption is masked. Further more we allow customers to recuperate the bulk of the costs though an oblivious cryptographic rebate system. Our technical report on differential privacy and rebates in metering is available.
• Finally proving that protocols are correct is not sufficient, so we explore options for proving actual implementation of the protocols are in fact providing the necessary security and privacy properties. A report on the certified implementation of a variants of the proposed protocols using refinement types is also available.

The project web-page on privacy in metering links to all those any more.

So much about the science, what about the engagement with government. On the positive side, our rather limited goal has been achieved: we wanted to put privacy technologies, that provide solutions beyond the dilemmas and balance between privacy and value, on the map. The government response to the consultation takes note, in a limited way, of the potential use of privacy technologies. On page 10 it shyly mentions that:

“2.18. Work is in process to understand the options for aggregating or anonymising smart metering data and whether it is necessary for the data to be accessed by a party that carries out the data minimisation. Privacy enhancing technology can potentially enable anonymised or aggregated data to be provided without any party having access to the personal data itself. The programme will work with industry and academics in order to explore the applicability of privacy enhancing technologies within the smart metering system.”

This is actually a rather fair representation of the capabilities of the technology, even if it is presented as a far away goal, rather than the concrete protocols we have proved correct and the implementations we have built.

Paragraph 2.18 mentioning privacy technology is a ray of light amidst an otherwise ambivalent government response. On the up side it recognizes energy consumption as private data from the onset, it mandates meters to hold 13 months of consumption and provide local access to it, it defines narrowly the scope of data that can be gathered without explicit consent and puts them under the data protection regime. On the down side there is confused language about what constitutes personal data (2.17), and the final technical solution involves collecting data in clear through a centralized systems (the glorious DCC) and protecting it using access control — a far cry from what we know possible in terms of technical privacy protection.

The metering privacy geeks (legal & technical) might also find other interesting nuggets in this report:

• It mentions privacy-by-design, but without support for privacy technologies (except a mention of aggregation in 2.14). This is a damaging trend set by the Ontario report on privacy in the smart grid that takes a purely management approach to privacy in the local smart grid deployment. A response to this trend is provided by Prof. Claudia Diaz and her colleagues that highlights the technical protections necessary to engineer privacy-by-design. This is only the start of this tussle.
• The report seems to suggest that personal data is not personal if it is not readily identifiable by the data controller (sect. 2.17 and 3.7). This is the classic argument of “what is de-identified personal data”. Does it mean the data controller cannot identify it, or anyone in the world? It seems the government is as confused as everyone else on this matter.
• The key outcome of the consultation is that the energy industry needs some data to perform “regulated duties”. This concept was present in the initial consultation, but funnily enough there was no description of that those duties were. It transpired in meetings that Ofgem was not in fact clear about what they were, and a large part of the consultation centered around fleshing those out. A list of those duties is available in Appendix 3 of the report, and is probably welcome by all (a similar list is available in the NIST privacy reports).
• So (in 3.15) the government concedes that industry must have access to the data necessary to perform its regulated duties by default, yet this data should be subject to the DPA requirements (3.16 for example specifically calls principle 5 — that the data should not be kept longer than necessary). Well that is a mine field: it is clear that the data is collected for a specified purpose (principle 2). If the other principles are also applied it means that it should not be used without explicit consent for other purposes (*cough*added value services*cough*) and furthermore it should not be excessive for the stated purpose. Well here we are: our technical reports offer ways in which most of the stated purposes in appendix 3 could be fulfilled without collecting the data. Is this a contradiction? Not automatically. The government’s view is clearly that our proposed protocols are not yet ready for prime time — of course as these technologies become better known and deployed this objection will evaporate. Will the data minimization requirement then mandate the use of privacy technologies? This is a rhetorical question at the moment.
• It is interesting to note that the restrictions associated with limiting the automatic collection of data by suppliers was possibly set in place on the grounds of market competition rather than privacy per-se (section 3.32). Automatic collection by suppliers would put them in an advantageous position vis-a-vis third-party providers of value added services. This is an open issue (3.36).
• The government is keen for a local repository of consumption data in the meter (4.6) and the use of geeky toys to visualize it (4.12). This is the setting in which our solutions enable strong privacy guarantees. That is positive, if only half-way.

In conclusion, the debate around privacy in metering has been informed by consumer concerns, privacy concerns, industry needs and technology alternatives. They are all represented in the government response. Yet the final solution is rather conservative: it relies on a centralised conduit for personal information protected by access control layers and management layers. It is far from what we know possible with privacy technologies. The argument today is that those technologies are too new — which is questionable given how quickly IT inovations are brought to market. This argument will lose its potency in the long term if we keep developping and deploying privacy firendly solutions.

BBC journalists, in their usual “impartial” style are quick to report the offence under which Mr Drake was arrested, but of course never convicted of. I will not be repeating it here as it might constitute slander, since the accusation was never in fact show to be true, and it is not even clear if the basis of the original suspicion played any role in the conviction.

The BBC also relays verbatim Det Sgt Neil Fowler, of Lancashire police, as saying: “Drage was previously of good character so the immediate custodial sentence handed down by the judge in this case shows just how seriously the courts take this kind of offence. [...] It sends a robust message out to those intent on trying to mask their online criminal activities that they will be taken before the courts with the ultimate sanction, as in this case, being a custodial sentence.”

Of course what the BBC’s impartial style fails to comment on, is that Mr Drake was in fact never shown to be participating in any online criminal activities aside the activity of not revealing his key to the police. At best it sends a robust message that innocent people mindful of their privacy in relation to the state will end up in jail, and at worse it signals to every serious criminal that if they do not reveal their keys they will get off with a light sentence. The police have powers to obtain warrants to enter premisses covertly, install surveillance equipment to retrieve keys, but instead they chose to simply ask the suspect to self incriminate themselves — this is poor policing, and will inevitably lead to travesties of justice.

This is just the beginning of RIPA part III being used, and of course I am looking forward to monitoring the legislation being used against people with legitimate needs for privacy, such as political activists, journalists, lawyers, whistleblowers, etc. Watch this space.

This is a very interesting paper on people’s attitudes to behavioural advertising. Researchers used a mix of a small-scale (14 people) study and a larger (100s of people) statistical study. A few findings are remarkable:

• First, they see that users apply their intuition of off-line ads to the experience of on-line ads — many see on-line ads as a push mechanism and do not realise that data about themselves are collected. They seem to not object in general to the idea of advertising, and consider it as a fact of life, and even see it as ‘ok’ to support services.
• The landscape of attitudes to behavioural advertising is fascinating. When faced with a description of what behavioural advertising collects, as a hypothetical scenario, and how it functions, a large percentage of users said this is not possible, and some of them even claimed it would be illegal. When it comes to attitudes towards receiving ‘better’ ads only 18% of them liked the idea for web-based services, and 4% for email based services (like hotmail & gmail). In general the authors found that a lot of extremely common practices cause “surprise”.
• The researchers also looked at the formulation of the text of the NAI site, that offers an opt out from behavioural advertising. They find that what the system does is unclear, even after reading the page where the operation is described.

In general people prefer random ads rather than personal ads, with the exception of contextual ads (like books on on-line book stores). There is still a lot of ignorance about how technical systems work, and education when it comes to privacy and the ability to self-help themselves to protect privacy is clearly not working.

This research is pointing in the direction that the presumed tolerance of users to privacy invasion is due to ignorance of common practices. Once those practices are revealed it produces surprise, and even feeling of betrayal that will not be beneficial to any company and customer confidence.