717036 - Jetpack Performance Tests

Status: RESOLVED WONTFIX

(Testing :: Talos, defect)

Description

[Jeff Hammel]

To run on checkin ala  https://tbpl.mozilla.org/?tree=Jetpack to see
if changes in the SDK has a performance impact.

The general procedure:

• Download a recent build of firefox
• Download the latest Add-on SDK
• Download the test add-on sources
• Build each test add-on with the add-ons SDK.
• Run the talos tests on Firefox standalone and then with each of the
  built add-ons installed.

A possible implementation:

* run talos with a set of tests: ts:ts_addon1:ts_addon2:...
** runs ts standalone, collects a set of numbers
** runs ts + addon1, collects a set of numbers
** runs ts + addon2, collects a set of numbers
* upload results to graph server (or custom server)

Comment 1

[Jeff Hammel]

So I have made a preliminary mozharness for jetpack and talos testing:

http://k0s.org/mozilla/hg/jetperf

This checkouts http://hg.mozilla.org/projects/addon-sdk , creates a "helloworld" addon, and runs talos ts test with it installed...maybe.

I have been trying to confirm that Talos, with `PerfConfigurator -ts`, actually loads addons, so I made a jetpack with a `main.js` similar to:

"""
const file = require("file");
exports.main = function(options, callbacks) {
var foo = file.open('/home/jhammel/foo.txt', 'w');
foo.write("i is a cat!");
foo.close();
}
"""

This works fine when running with "mozrunner -b `which firefox` -a /home/jhammel/mozilla/src/jetperf/src/helloworld/helloworld.xpi".  However, running

"""
python PerfConfigurator.py -a ts -e `which firefox` --extension /home/jhammel/mozilla/src/jetperf/src/helloworld/helloworld.xpi --develop
"""

and the subsequent `run_tests.py` command, a '/home/jhammel/foo.txt' does not appear.  I am not sure what is going on here.  Maybe this isn't possible with ts currently?  I will try to figure out if extensions are getting installed and why the helloworld.xpi code isn't hit.

In the meantime, if anyone wanted to look at my jetperf code and give feedback, I would appreciate it.

Comment 2

[Jeff Hammel]

We should also figure out when we want to run these tests in production: on http://hg.mozilla.org/projects/addon-sdk change? on http://hg.mozilla.org/mozilla-central changes? Both? other?

Comment 3

[Jeff Hammel]

wrt comment 1, just tried with `PerfConfigurator -a tsvg [...]`; same results; the addon doesnt seem to get called.

Comment 4

[Jeff Hammel]

Noting the Talos failure in a separate bug

Comment 5

[Jeff Hammel]

We should also figure out what to do about processes still running for Firefox.  Currently, we go through all the steps and will err out if talos finds processes still running.  Maybe we can/should do that first, though IMHO we should solve this in Talos to a larger degree.  Its good for production (though the way we do it is awful) but horrible for development.  Should file a follow-up bug for doing this with --develop though that needs to get defined a bit

Comment 6

[Jeff Hammel]

We should also figure out what numbers we are calculating and how we want to display them.

Lets say ts takes x time and ts + a jetpack takes 1.01x time.  Already, you notice, we want to separate a small differential from a noisy signal.  One solution to this is to install the same addon many times which will amplify (linearly) the differential.

What do we do with the numbers? Presumedly we want to upload them to *a* graphserver.  But how do you calculate what is "good enough"? We're dealing with a lot of this with https://wiki.mozilla.org/Auto-tools/Projects/Signal_From_Noise currently, but I would love to know what the JetPack team would want and flesh this out a bit.

Comment 7

[Jeff Hammel]

Ideally, we would measure from when the addon starts getting loaded to when it stops.  Currently we are just piggybacking on ts which just measures the onload event.  I'm not sure if we actually have events for this, or if what I'm doing is "good enough".  Insight welcome

Comment 8

[Joel Maher ( :jmaher ) (UTC -8)]

I would think that the purpose of this test is to see what impact we have on startup performance not the raw measurement of the addon startup time.

Comment 9

[Jeff Hammel]

Very preliminary results from http://k0s.org/mozilla/hg/jetperf/file/dc6f4e782841/jetperf/compare.py :

(jetperf)│compare ~/before.txt ~/after.txt 
/home/jhammel/before.txt: 
 -ts: 386.7
 -ts_shutdown: 577.15
/home/jhammel/after.txt: 
 -ts: 423.8
 -ts_shutdown: 604.25

before is the data with the plugin not being loaded, after is the data with the plugin being loaded

Comment 10

[Jeff Hammel]

Attachment: graphical output

Comment 11

[Jeff Hammel]

JetPerf now runs ts side by side with the addon and generates graphical output with the --graph-output flag.  Try it!

Comment 12

[Jeff Hammel]

see also https://wiki.mozilla.org/Auto-tools/Projects/JetPerf

Comment 13

[Jeff Hammel]

Attachment: real numbers

So these should be some real numbers with just `require("panel");`

/home/jhammel/mozilla/src/jetperf/build/results.txt: 
 -ts: 814.15
 -ts_shutdown: 211.65
/home/jhammel/mozilla/src/jetperf/build/baseline.txt: 
 -ts: 717.75
 -ts_shutdown: 182.9

Comment 14

[Jeff Hammel]

Just for fun, I ran with 50 addons (all just `require("panel");`):

(jetperf)│compare-talos baseline.txt jetperf.txt 
filters: mean
baseline.txt: 
 -ts: 401.05
 -ts_shutdown: 612.45
jetperf.txt: 
 -ts: 2444.7
 -ts_shutdown: 1286.1

Comment 15

[Jeff Hammel]

So http://k0s.org/mozilla/hg/jetperf is at what I would consider beta stage, so I encourage stakeholders to look at it, see if it does what you need, ask me questions about it, and use it.  I will work in the background about polishing up nits, but unless I hear otherwise I will assume that the basic structure is what we want and go ahead with getting buildbot and graphserver set up for this.

In short now is the time to decide if this is the basic architecture we want initially and for any medium term future!  We can easily change it now....once infrastructure is in place we won't be able to so easily and the form of this software largely determines how buildbot and graphserver should be set up.

Comment 16

[Dietrich Ayala (:dietrich)]

Hi Jeff, I haven't had a chance to look at the code yet.

Is there a spec or documentation that describes it's intended use-cases and current capabilities?

More specifically, can I do something like this:

Use jetperf to run Test A (any talos test: ts, tp5, txul, etc) with build B (any build that have locally) with add-ons C (where C is one or more add-ons).

Comment 17

[Jeff Hammel]

(In reply to Dietrich Ayala (:dietrich) from comment #16)
> Hi Jeff, I haven't had a chance to look at the code yet.
> 
> Is there a spec or documentation that describes it's intended use-cases and
> current capabilities?
> 
> More specifically, can I do something like this:
> 
> Use jetperf to run Test A (any talos test: ts, tp5, txul, etc) with build B
> (any build that have locally)

Yes and yes.

>  with add-ons C (where C is one or more
> add-ons).

Currently you can run with any single-file (main.js) jetpack (cfx).  If we want something more than that, that should be ticketed and we should decide on a list of what we want for production. See bug 717980

Comment 19

[Jeff Hammel]

Attachment: jetperf now hangs

I've noticed today that the pagemod-addon no longer works with talos tests.  I suspect either a change in the addon-sdk code or in Firefox.  Running

talos -n -d -e `which firefox` --develop -a ts --extension /home/jhammel/mozilla/src/jetperf/src/mozharness/build/addons/pagemod-addon/pagemod-addon.xpi

each cycle (== browser restart) hangs at resource://pagemod-addon-at-jetpack/pagemod-addon/data/document.html with the output given in the attachment.  I can't see any reason for this in the error console or the developer console.  But something has changed.

Comment 20

[Jeff Hammel]

This does not occur with either empty-addon or widget-addon from http://hg.mozilla.org/projects/addon-sdk-jetperf-tests/file/d7e90684309c

Comment 21

[Alexandre Poirot [:ochameau]]

It doesn't seems to be due to Firefox as it fails with Nightly and current stable release. (I'm able to reproduce it) I'd say something collide between some SDK code and some talos code.

Do you know which code in talos do close Firefox ?

Comment 22

[Alexandre Poirot [:ochameau]]

I have no idea how talos really works but isn't it because it is trying to load the following html file:
http://hg.mozilla.org/build/talos/file/7d28f2533eb5/talos/startup_test/startup_test.html
But in case of page-mod, this page will be immediatly unload and replaced by page-mod addon page. So that this html page won't have time to execute till its end and especially not this quit action:
39:    goQuitApplication();

Comment 23

[Alexandre Poirot [:ochameau]]

That's it. It works if you replace, in pagemod-addon/main.js:
require("addon-kit/tabs").open(testURL);
by:
require("addon-kit/tabs").open({
  url: testURL,
  inNewWindow: true
});

But it is not ideal as we are opening a new window. It will add a perf cost that is not directly related to a jetpack action. Ideally, talos should work with this code:
require("addon-kit/tabs").open({
  url: testURL,
  inBackground: true
});

Comment 24

[Jeff Hammel]

ticketed: https://bugzilla.mozilla.org/show_bug.cgi?id=768215

Comment 25

[Jeff Hammel]

Sorry its taken so long to get back to this bug.  There were numerous
Talos and Datazilla-related issues that needed to be tackled before I
could do much here.

The current chief tasks are:

- get jetperf to post to datazilla. This is blocked by the deployment
  of passwords.py on the relevent build slaves, but as far as I know
  that is it.

- enable jetperf on buildbot.
  - staging

Jetperf will post its results to datazilla.  For those that don't
know, datazilla is the new graphserver replacement for accumulating
performance data.  Jetperf results will be seen at
hhttps://datazilla.mozilla.org/jetperf/ . Some manual inspection will
initially be necessary to see if jetpack changes cause a performance
regression.

Since one is comparing the small differences in numbers running with
and without cfx-generated addons, it remains to be seen how difficult
it is to separate the signal from the noise.  Jetperf runs two tests:
1. a baseline
2. with addons enabled.

The graph at
https://bug717036.bugzilla.mozilla.org/attachment.cgi?id=592916 seems
to indicate, minus the first point, that the signal is measurable.
However, datazilla will not generate charts in this manner, at least
initially (and possibly ever).  The driving force for datazilla has
been Talos, and specifically detecting regressions from one
mozilla-central change to the next.  Datazilla is still in the early
stages of being able to do this. Jetperf's driving case is a bit
different.  We want to see two things with jetperf:

- if a change in mozilla-central causes a regression in
  jetpack-generated addons

- if a change in the jetpack repository causes a regression in
  jetpack-generated addons.

While in theory one could get away with not running the baseline test,
the existing infrastructure is probably insufficient at this point to
make that a reality.  In addition, for mozilla-central changes, the
naive implementation (comparing one changeset to the next) conflates
a regression in Firefox's performance with that in the addon-sdk's
performance.  So it is necessary for now to run a baseline comparison
along with addon benchmarks.  In the future, we should reconsider this
when a path forward becomes clear.

As far as I know (:jeads?) datazilla has a full RESTful API, so if we
need a dashboard to make this information more decipherable, that
should be doable.  I think deploying first and then assessing the
situation makes a lot more sense than trying to figure out all of this
before we have data coming in, however.

Another thing that should be considered is how to run tests.
Currently, all addons in a particular bundle (
http://hg.mozilla.org/projects/addon-sdk-jetperf-tests/ ) are run.
I've written bug 770690 that proposes that one addon at time is run,
at however a greater cost of run time.  Neither the Jetpack team nor
RelEng have weighed in on this bug, so I'm not sure if this is a
priority and/or a good idea or if there is enough slave time to go to
e.g. double our run time for jetperf.

Which leads us to the final issue: actual deployment (bug 720901). I'm
not sure who's head this is on, though I'm guessing mine.  I'd
appreciate any guidence in figuring out what needs to be done here.
Otherwise I'll try to figure out what needs to be modified.

Comment 26

[Joel Maher ( :jmaher ) (UTC -8)]

is this still a valid bug?  Do we already run jetpack performance tests?  I want to figure out what needs to be done with this bug if anything at all.