The decision must have been a tough one, most especially when there was a meta review (review 4). I am accepting the comments, which will help me improve the paper. Below is the notification I got on the paper. I hope to revise the paper soon and submit it to another conference.
From: submissions2010{at}iptcomm.org
Subject: [IPTComm'10] Your paper #1569296037
Date: Sat, May 22, 2010 1:30 am
To: ”Michael O. Adeyeye” <micadeyeye{at}crg.ee.uct.ac.za>
Cc: vkg{at}bell-labs.com,Gonzalo.Camarillo{at}ericsson.com
Dear Mr. Michael Adeyeye:
We regret to inform you that your paper #1569296037 entitled ‘Quality of Experience
of the HTTP Session Mobility Service’ was not accepted for IPTComm 2010.
This decision was not easy to make, but we took into consideration the reviewers’
comments as well as the overall theme of the conference. We hope that you will find
the attached remarks constructive and helpful in improving your paper. We look
forward to having an enhanced paper or any of your future work in the upcoming
IPTCOMM conferences.
The reviews are attached below. You can also find them at your EDAS personal page.
Best regards,
Vijay K. Gurbani and Gonzalo Camarillo
IPTComm 2010 TPC Co-chairs
======= Review 1 =======
> *** Contributions: What are the major issues addressed in the paper? Do
you consider them important for IPTCOMM? Comment on the degree of novelty,
creativity and technical depth in the paper.
The authors present a proxy-based service for sharing content between two (or more)
users via their web browsers. (This makes the title misleading since session
mobility is something else.) The authors choose to use SIP messaging (via the
MESSAGE method) to realize content sharing between users, allowing to pass
references. They implement some of the service as SIP servlets in a server.
The authors briefly (well: too briefly) outline the operation; not all the signaling
is covered; e.g., suddenly a conference bridge appears for forwarding streaming
data, but it is unclear where this comes from.
> *** Strengths: What are the major reasons to accept the paper?
The topic of shared browsing (while not new) is interesting and still relevant.
The authors provide a complete implementation of their ideas.
> *** Weaknesses: What are the most important reasons NOT to accept the paper?
The paper contribution is unclear. Apparently, the authors have published
significant parts of the system before [4, 10] — which seems to be the session
mobility part, not the shared browsing. However, the delta does not become clear;
especially with respect to the evaluation.
The authors evaluate the performance of their server — but it is unclear if this is
now specific to the sharing extensions presented here or just a performance report
on their earlier work.
It is also questionable if the evaluation serves the purpose: yes, understanding the
load implications on the server is meaningful, but what does this say about quality
of experience.
What is actually the experience of a shared session? What does sharing ‘session
data’ actually mean. The paper is too imprecise in many of these technical respects
and thus the specific design cannot be fully understood and appreciated.
> *** Detailed comments: Please provide detailed comments that will be helpful to
the TPC for assessing the paper, as well as feedback to the authors.
Fix the title to better represent what the paper is about.
Don’t make claims (“report on Quality of Experience”) that you do not fulfill.
Watch your accurary: 5 digits are way to much presumed precision for the values
presented.
How would extending the service beyond two browsers scale? How would it be
implemented?
> *** Relevance: Relevance to IPTCOMM
Definitely relevant (4)
> *** Familiarity: Rate your familiarity with the topic of the paper.
Familiar (3)
> *** Overall rating: Your overall rating
Weak Reject (2)
======= Review 2 =======
> *** Contributions: What are the major issues addressed in the paper? Do
you consider them important for IPTCOMM? Comment on the degree of novelty,
creativity and technical depth in the paper.
This paper describes a HTTP session mobility service wherein HTTP session data are
transferred between Web browsers as payload of SIP MESSAGE requests, which in turn
are transmitted through a SIP network. Much of the system has already been
described in Ref [4]. This paper describes the network-resident Converged
Application Server (CAS) in more details, and provides performance evaluation of the
CAS. The paper also describes a ‘Stream Media to Call’ service.
> *** Strengths: What are the major reasons to accept the paper?
The CAS is implemented using a dominant programming API standard (SIP Servlet) and
an open source SIP Servlet container (Mobicents SIP Servlet Container). Therefore
the performance evaluation result should have wide interests from practitioners and
researchers in the field.
> *** Weaknesses: What are the most important reasons NOT to accept the paper?
It appears that the HTTP session mobility service has already been described in Ref
[4]. The additional contribution in the submitted paper is mainly performance
evaluation of the network-based application (CAS). However, much of the paper is
spent describing the system at the expense of a more extensive discussion and
analysis of the performance evaluation.
The paper aims to present the ‘quality of experience’ of the system. However, it
seems the metrics used, i.e. CPU usage and memory consumption at the proxy of the
system, is not a good measure of quality of experience as perceived by the users.
These metrics are more relevant to service providers planning a deployment of the
service. The paper does not provide sufficient justification of the choice of the
metrics, e.g. relevant citations, or subjective tests that show strong correlation
between quality of experience as reported by the test subjects and these metrics.
From a performance evaluation standpoint, the choice of experiments was unclear.
Using bursts of various number of messages arriving at a fixed overload rate does
not seem to model actual traffic pattern. It would seem for an application server
in the network serving large number of users, message arrival rate would be rather
uniform?
In this reviewer’s opinion, this paper does not present sufficient information on
the research challenges, tradeoffs and solutions faced during the design and
implementation of the CAS. The analysis of the performance evaluation was also
limited. A number of areas in which this paper may be enhanced before publication is
listed in the Detailed section below.
> *** Detailed comments: Please provide detailed comments that will be helpful to
the TPC for assessing the paper, as well as feedback to the authors.
In III, first para: “The proxy is a Converged Application Server (CAS) that
responses to both SIP and HTTP requests. It acts as a SIP Back-to-Back User Agent
(B2BUA) between the interacting browsers.”, and from III.B. and Figure 3 it appears
that CAS acts as a SIP proxy. The reader is confused whether CAS acts a SIP B2BUA or
a proxy, which have precise meaning in SIP. As the performance characteristic may be
very different, this should be clarified. It would seem the CAS function may be
fulfilled by a SIP proxy, so if it is a B2BUA it should be explained.
III.B To make the performance evaluation useful to readers, should specify which
version was tested and whether it was the Tomcat or JBoss version.
III.C While the ‘Stream Media to Call’ service is interesting and may have many
applications, it is not clear how it is relevant to HTTP session mobility. Perhaps
this service may be the subject of a separate paper.
IV. What is the message arrival rate in the ‘normal load’ case (Table 1)? How does
the CPU Usage varies with different rates? What is the maximum sustainable rate,
say at 80% CPU usage? This would be helpful to give a better indication of the
performance of the implementation. This would also help put the 677 requests/sec in
overload condition in perspective.
Some statements in IV are surprising and should be explained in more details. e.g.
Why is the CPU usage at 54% when CAS is idle? What causes the very large 10s delay
when over the Internet? Table 1 shows average time for CAS to send a 403 response
is near a second. And there are timeouts even when CPU Usage is 20%. What are the
cause?
Some additional discussions that may add value to the paper may be:
- Compare the message pattern in the HTTP session sharing service against a regular
VoIP service. Are there any differences in the traffic characteristics, e.g.
burstiness, message size, etc? How would these differences if any influence the
design and implementation of the CAS?
- How does message latency affect the quality of experience in this service? Is the
expectation of responsiveness and latency different from VoIP calls?
- What lessons were learned in the implementation of CAS? What design tradeoffs
were made? How can performance be improved. e.g. use of relational database server
to store user profiles vs in-memory database?
> *** Relevance: Relevance to IPTCOMM
Definitely relevant (4)
> *** Familiarity: Rate your familiarity with the topic of the paper.
Familiar (3)
> *** Overall rating: Your overall rating
Borderline (3)
======= Review 3 =======
> *** Contributions: What are the major issues addressed in the paper? Do
you consider them important for IPTCOMM? Comment on the degree of novelty,
creativity and technical depth in the paper.
The major issue addressed in the paper is how to share an HTTP session using SIP.
The idea is sound and quite innovative (other systems for moving bookmarks and
sharing URLs are available today but this work tries to go beyond them addressing
real-time-constrained issues related to HTTP session mobility). The authors have
indeed had creativity here. The technical depth could have been enhanced by
providing session mobility measurement in realistic scenario instead of
demonstrating only the performance of the implemented software.
> *** Strengths: What are the major reasons to accept the paper?
The idea is nice, the authors demonstrate to have creativity, the code used for the
experiments is made available open source. The concepts could be further exploited
for a more social-oriented usage.
> *** Weaknesses: What are the most important reasons NOT to accept the paper?
The quality of experience metrics are poorly defined (I would have expected
something more than CPU usage and memory consumption). The paper is much “service
oriented” without addressing the networking issues of this topic that could make the
overall solution not usable (delays, etc.).
> *** Detailed comments: Please provide detailed comments that will be helpful to
the TPC for assessing the paper, as well as feedback to the authors.
I have no particular comments to the paper as it seems to me well written even if in
a basic level of English. Scientifically the type of experiments could have been
better performed.
> *** Relevance: Relevance to IPTCOMM
Definitely relevant (4)
> *** Familiarity: Rate your familiarity with the topic of the paper.
Familiar (3)
> *** Overall rating: Your overall rating
Strong Accept (5)
======= Meta review 4 =======
> *** Contributions: What are the major issues addressed in the paper? Do
you consider them important for IPTCOMM? Comment on the degree of novelty,
creativity and technical depth in the paper.
The authors present a system that allows sharing HTTP-based by means of SIP. They
have implemented their system and provide an implementation-based evaluation.
> *** Strengths: What are the major reasons to accept the paper? [Be
brief.]
+ interesting idea, some novelty
+ working prototype implementation
+ implementation-based evaluation
> *** Weaknesses: What are the most important reasons NOT to accept the paper? [Be
brief.]
- paper does not include the promised/claimed QoE evaluation
- delta over earlier work unclear
- usefulness of the evaluation questionable
