Trying to wrap an answer to the above threads:

The key question is what are we trying to achieve:

1) High quality image database
2) High traffic and therefore revenue and profitability

The two are interrelated and this is what the a.net" brand has come to stand for. Point 1 : High quality image database. Therefore, it is critical that we maintain that consistency, even if sometimes some "great" photos do not make it to the database. But remember, a lot do, but uploaders only remember those which didn't make the cut, that's human nature. If you say that 3 positive comments from screeners should be enough, rather than a 4:3 vote, then where does the line end, maybe 2 is enough, perhaps one day, 1, like Johan used to be the final arbiter.

Paul, regarding your question on "appraisal process" no there is not one. But I know for a fact that all screeners actively look at the appeal email that goes out, showing what was or was not successful on appeal, to learn and calibrate better.

And we actively monitor screener standards, if we see deviations from what is supposed to be acceptable, then we will step in and inform the screener. It's a pretty good system. So in an informal way, there is an appraisal process.

Regards

Andrew

Quoting il76 (Reply 41): I often 'direct added' shots that were a bit more 'outside of the box' and where I felt a risk that it might get rejected by another screener who would 'go by the book' if I left it in the Q for second screening. Stuff like extreme panning shots not being 100% sharp, or creative framing/centering (e.g. Rule of 3rds). It was a really good feeling if you'd see it do well in the ratings the next day.

Ed, I can totally see myself being this type of screener, if I ever was one in the future. I think I would get fired pretty quickly because I would just look at something for the beauty of it, and accept it. Of course, poor editing, poor photo, reject. But I would simply look at the viewer appeal factor, myself included, and accept it. The next day I would be searching the DB for those special photos I added hoping the viewers approved of my screening work! I think I would give the Heads more headaches than help! lol  

Crew members - very nice thread and responses to the OP. Thank you for your contributions here and of course to the process that goes on behind the scenes. I'm not always happy with everything here (and don't have to be), but I know all is done to better the site from your viewpoints, and you are all passionate about this hobby. My hats off to you, thank you for your hard work.

I thoroughly enjoy the interactions here between crew and contributors, I hope it continues in the future and translates into the screening process, and we continue to build relationships and foster collaboration.

Thanks!

I've read all this interesting thread... I have to say that it has answered to several question on the whole process that I had in my mind.
So, first of all, I'd like to thank the crew members that have shared their "behind the scenes" in order to make all clearer and transparent!  

then, if possible, I have a further question: do the screeners take into account the difficult of taking certain kind of shots (pannings in poor lighting conditions, for example)? Or do they just evaluate in an objective way what they see?
I think it would be a kind of melting of the two, although always in keeping with the standards of the site... but I can be wrong...

Thanks in advance,

Matteo

Quoting Psych (Reply 48): Hello Andy - this could be seen as a justification for not rejecting a submission at its first screening.

If you're the first screener to see a shot that has an obvious flaw (e.g. dirty or unlevel) why wouldn't you just reject it straight away? There's no need to waste another screener's time if the photo is obviously going to be rejected. Equally, as has been mentioned above, once a screener has sufficient experience, they're encouraged to "Instant Add" shots which are clearly good enough.

Thus we have screeners rejecting and accepting photos on their first viewing. IMHO this is a good thing and helps increase queue throughput.

Hi all,

I've pretty much stopped posting in the forums these days (not due to the screening process) but still like reading the odd thread now and again. I felt compelled to comment here as I feel the screening has found a little more consistency of late, particularly concerning the centre rule.

Whether the subjective issue of centring has been discussed behind the scenes I don't know but I have to confess to altering my centring preferences with some of my recent shots, just to test the water. I have been very pleasantly surprised if I'm honest, as I thought the problem would just intensify with even more screeners on board.

Logically it would figure that the more people involved, the greater the range of opinions; I guess time will tell but so far, so good.

Unusual for me to have nothing to moan about but I'm sure I'll find something soon.....  

Karl

Quoting teopilot (Reply 54): I have a further question: do the screeners take into account the difficult of taking certain kind of shots (pannings in poor lighting conditions, for example)? Or do they just evaluate in an objective way what they see? I think it would be a kind of melting of the two, although always in keeping with the standards of the site... but I can be wrong...

Usually yes, unless the flaws are severe. Low light panning shots are a good example - more grain/noise and a bit more softness is usually tolerated.

Quoting scbriml (Reply 55): If you're the first screener to see a shot that has an obvious flaw (e.g. dirty or unlevel) why wouldn't you just reject it straight away? There's no need to waste another screener's time if the photo is obviously going to be rejected.

The reason I sometimes will ask for a second opinion on a shot with a clearly rejectable flaw is to get another opinion on another potential rejection reason. For example if there is a dust spot, but also a questionable motive, I will ask for a second opinion, pointing out the spot. That way if others agree, the rejection is inclusive, and the photog won't fix the spot, re-submit and then have rejected for a different reason. Otherwise, yes, obvious flaws are rejected at first viewing.

Thank you all for a terrific thread... very insightful. Since I have your attention, I'd like to ask about the screening itself... the subjectiveness vs. technical (Dana, you know me   I love this topid so I had to ask)

I tend to get in to trouble a lot with Centering, Color and Brightness and also sharpening butthat's I think due to the time I edit sometimes LOL.

Color and Cast etc: I usually argue with your rejections (to myself, at my computer, rarely here) because I will have gone over the NEUTRALITY of the image 100 times. Typically I make sure the whites, grays and blacks are neutral, that is the same or close level for Red Green and Blue. While not perfect, it is close. However I get rejected and have no recourse but to guess at what would be acceptable. I HATE having to post in this forum for each image. I really do try my hardest to get it right on my own the first time. So, does technically correct ever lay a role? I know the interface may not be able to help and certainly not by bringing images into photoshop. But perhaps a tool could be devised to simply hover over the image on neutral colors and check for neutrality in whites/blacks/grays
As for Contrast, I tend to stick to the reality of the scene and a lot of times, the bellies ARE THAT dark :P

Overexposed: I do agree 99% of the time with my dark rejections. But when it comes to overexposed, that's tough for me to deal with as I usually check the whites and/or brightest colors to be within a luminosity of 250 or less at 8 bit or 90 out of 100 if using a scale of 100 for pure white. Yet I still get rejections of overexposure and then struggle with re-editing because any less exposure makes the image look un-natural 

Sharpening: I still step over the line from time to time but after Dana schooled me graciously in a really great off-line session, I now know more about A.net sharp than ever before... At least for me, I am very grateful for that.

Heat Distortion from Engines: I never know when it will be recognized by a screener as that or Heat Haze from distance/Sun heat. IF shooting the back of a jet and the engines distort the aircraft, is that instant reject or is it subjective? I assume subjective but then it puts me back to square one with my indecision to up or not hehehe.

Thanks guys. I have a ton of questions but I think the above is enough for now... Oh one more:

Female Screeners? The hobby is predominantly male but I do know a few superb female enthusiasts.

..Manny

So, I vote for more technical considerations in screening

Quoting gonzalu (Reply 58): Color and Cast etc:

Quoting gonzalu (Reply 58): Overexposed:

These things (as with all other technical aspects) will always be determined by eye. Not only do we not have the ability within the screening process to assess such criteria other than by eye, but taking the time to measure each of those criteria technically would add a lot more time to the screening process. Relying on human judgment of course means there will be mistakes from time to time, but I don't think I'm going out on a limb if I say we get it right a vast majority of the time. My own personal belief is that if we started requiring tools be used to measure color/contrast/exposure and determine acceptance criteria, it would put those than cannot avail themselves of such tools at a great disadvantage.

Quoting gonzalu (Reply 58): Heat Distortion from Engines: I never know when it will be recognized by a screener as that or Heat Haze from distance/Sun heat.

Heat haze is not the same as jet wash, and we all have enough experience to know the difference. If heat haze is judged to be an issue, it will only be based on part of the aircraft that couldn't possibly be affected by jet wash (i.e. forward of the engines). Jet wash will only be a problem if it is obscuring a majority of the aircraft, but exactly how much will be a subjective call.

Hello

I also screen and monitor the 'Re-Upload' page.

A lot easier to screen as no information is to be checked. Basically just compare the new upload with the current database one. Accept or reject. Not an automatic acceptance. Sometimes see worse re-uploads.

Approximately 25-30 per day and I like to clear daily.

This page/process also has a bug. Sometimes the large image is not replaced but thumbnail is OK. I have a 'Purge' page to select the offending image and re-process again. Usually that works.

Regards

Gary Watt - Granite
Airliners.net Head Screener
www.airliners.net
http://twitter.com/airliners_net

Quoting gonzalu (Reply 58): Overexposed: I do agree 99% of the time with my dark rejections. But when it comes to overexposed, that's tough for me to deal with as I usually check the whites and/or brightest colors to be within a luminosity of 250 or less at 8 bit or 90 out of 100 if using a scale of 100 for pure white. Yet I still get rejections of overexposure and then struggle with re-editing because any less exposure makes the image look un-natural

This was always a part of the screening process that I thought should warrant more eye-based judgement as opposed to an over-reliance on histograms. Dana alludes to the fact that determining over-exposure is often done using the naked eye, but I imagine at times there is too much faith placed in the histogram. Trouble with histograms is they aren't always accurate. Common tricks of light (sun reflecting off parts of the aircraft is a good example) can easily fool the camera into producing a 'perfect' histogram, yet in reality the exposure is out by quite a bit. Similarly, the histogram can hint at an incorrect exposure when in fact it is spot on to the eye.

In our line of photography, we have to aim to get the subject (aircraft) properly exposed, while the background is much less of a worry. Seeing as a histogram evaluates a whole scene and not just sections, it's not always a good idea to rely entirely on a histogram.

Karl

Quoting gonzalu (Reply 58): So, I vote for more technical considerations in screening

Screeners use as their only tool their eyes, anything else just takes way too much time. I often consider that while editing a complicated picture may take quite some time, it will be judged by the screeners literally within seconds. In other words, it has to look right (in the screeners eye through their monitors).

I think that is probably the most challenging part of training new screeners (and it happened also to me in the beginning), you really need to train your eyes. By doing this over and over again hopefully every day, you start building up your confidence in your eye-judging qualities until you are ready to look at a picture and maybe spend 15-20 seconds looking and judging it (then you'll need another 10 seconds or so to tick boxes, and then it's up to the next one). That is the only way you can screen maybe 100 picture in one hour. Indeed it will take some time (and many thousand pictures screened) until any screener can develop this speed coupled with the accuracy of his judgement.

While it may look as too little time, overall the results are very good (as we can see every day from the accepted pictures), and for the few errors there is the appeal function.

Cheers,
Andres

Karl

Quoting JakTrax (Reply 61): Dana alludes to the fact that determining over-exposure is often done using the naked eye, but I imagine at times there is too much faith placed in the histogram.

You have misinterpreted what Dana said - It is not often done by eye, it is always done by eye.
At a very early stage in this thread I showed the screening page and that is what we get, no other tools, no histogram, no equalising, not even any cursors, we check levels by scrolling to the edge of a window.
We do not, we cannot check the histogram at all other than by saving it and then opening in Photoshop and we do not do that.

When training new Screeners, one of the most difficult things to get them to accept is just how effective the circling method is for detecting dust spots and halos and I tell them to do just that, to save and equalise just until they learn to trust their eyes and the circle method after which they stop as it is very frowned upon detecting them that way. If they cannot be detected with the naked eye they should be accepted but we do see a lot !

Mick Bajcar

Different screeners have different standards. There're standards, there are no standards. This is airliners.net.

Quoting dendrobatid (Reply 63): You have misinterpreted what Dana said - It is not often done by eye, it is always done by eye.

Indeed, since I started screening never have I based any screening decisions on what I saw in the histogram. In fact, I'm pretty sure I have never even looked at the histogram of an image during the screening process. As Andres said, with some experience, our eyes are just as reliable (or even more so), and a lot faster.

Quoting JimmyLWH (Reply 64): Different screeners have different standards. There're standards, there are no standards.

Not quite sure what that is supposed to mean, but as individual screeners we do our best to uphold and follow the same standards equally.

Quoting il76 (Reply 45): Totally correct Andy, and that's why I'm not a screener anymore.

Been wondering looking at Eduard's account...

OK, the screeners don't get paid - it is a known fact.

Another known fact is the tremendous amount of work they have to put into screening and lots of their free time they have to sacrifice.

So, in the end, when they pull out for whatever reasons, don't they even deserve a free First Class Membership as a bit of a bonus for all their hard work?

Surely wouldn't cost DM a whole lot of money.

From the discussion in the beginning of the thread, I gather there are no precise stats on screening process (percentage of the images accepted, rejected, SO'ed, HQ'ed, etc) readily available.
Assuming screening results are kept in the DB, getting this info should be possible with fairly straightforward sql query. I understand there may be very good reasons not to publish numbers on the front page, but is someone monitoring numbers to assure screening consistency across the team? I mean, statistically, number and percentage of rejects for , for example, "Soft" should be roughly equal among all screeners.

Adding my THANK YOU to screeners for their dedication patience and long hours.

Quoting SFO2SVO (Reply 67): From the discussion in the beginning of the thread, I gather there are no precise stats on screening process (percentage of the images accepted, rejected, SO'ed, HQ'ed, etc) readily available.

My estimates were rough, but they are very close, within a few percentage points anyway.

Quoting SFO2SVO (Reply 67): I understand there may be very good reasons not to publish numbers on the front page, but is someone monitoring numbers to assure screening consistency across the team?

It is possible to see the individual rejection stats for each screener, but I didn't feel like compiling a precise number. Needless to say, the stats I quoted above are more or less accurate, though they will vary from screener to screener and day to day. As an example, newer screeners rarely Instant Add images, but more experienced once may do it for 20% of the images they see or more, which accounts for the average of 10%. Checking today's stats, I have in fact IA'd 19% of the images I've screened, but have rejected more than the ~50% average. Tomorrow it will likely be different. If you look at the numbers I posted above, you will see the overall average of rejected/accepted is 50/50.

Andrei

All screeners receive First Class Membership. This has always been the case since it was set up.

Regards

Gary Watt - Granite
Airliners.net Head Screener
www.airliners.net
http://twitter.com/airliners_net

Gary, I always knew that the screeners received First Class Membership. I just didn't know if they got to keep it if they stopped screening. Kind of assumed they would, even if just as a token of appreciation for all the hard work they would have have done. Then, looking at the account of Eduard Brantjes aka il76, I see that he holds just a Premium membership, and he was a screener once.

I realise that this sort of decisions is most probably out of the screeners' hands anyway. It was a rhetorical question directed more at the DM's management.

Without revealing anything proprietary or sensitive, could you walk us through a typical screening? What I mean is something like:

Open up the browser.
Log in to A.net (if not already cached)
Open the queue
Open a batch
Check for Brightness/Exposure
Check level
Check Sharpness
Check data
etc.

Also, on a more technical note, this bothers me because I know it is a huge problem in the real word, not sure if it affects you. Color Profiling and Monitor Calibration, Browser support for such and related.

I edit on a Mac Pro with a Apple 30" Cinema Display, hardware calibrated once a month or so. The browser of choice is Firefox 10.x with color profiling enabled. I typically convert my images to sRGB but even then, A.net removes ALL metadata from images, INCLUDING profile information. Since that's expected, I convert my images to sRGB so there is no chance of misinterpretation.

Given the above, are there some requirements for screeners to have their computers calibrated and have some minimum hardware requirements? Is there a browser of choice? Some browsers are terribly color profile brain dead. You can imagine how one screen being a little hot could cause an otherwise correctly exposed and processed image to look overexposed on a different screen.   happens to me at my work computer which is just horrible (sorry, we get not-so-good monitors at work   )

Thank you again for this great thread!

Andrei,

Quoting granite (Reply 69): All screeners receive First Class Membership.

...for a year, after you quit being a screener. After that you fall back to what you had before you became screener. Premium Membership in my case. No biggie...

Ed

I see this in the upload queue statistics:

2012-02-20: 1 photos 1 photos left for the Head Screeners
2012-02-24: 4 photos 4 photos left for the Head Screeners
2012-02-25: 1 photos 1 photos left for the Head Screeners
2012-02-26: 734 photos 629 photos not screened yet 105 photos currently in screening

If you will accept the picture which left from 2012-02-20 can you show us?
I would to know how looks like a picture which has got so long time in the queue.

Quoting eggohoek (Reply 73): If you will accept the picture which left from 2012-02-20 can you show us?

To be clear, the photo was not in the HS queue since the 20th, but it was in the regular queue so that all the team could see it and offer an opinion. It was passed to the HS queue today, where it was rejected for motiv. While we can't show you the photo I will tell you that there was a clear obstruction and the photographer already had two shots in the database of the same subject.

What is the optimal number of:

a) Screeners
b) Acceptances per day

In order to both keep the queue wait time at a reasonable level while keeping a high standard.

Does there become a tipping point where either there are too many screeners or where there are too many acceptances per day resulting in lesser exposure to those shots that are accepted?

In a perfect scenario how many days turn around would the screening team like to see between a photo being uploaded and either being accepted & rejected.

Over the winter the average screening time seems to be around the 6-7 days mark which is not too unreasonable given the current number of screeners and photos in the queue.

I think 5-7 days is "about right" for queue length, but that is just my opinion. When the queue is at that length we average about 1,500 uploads per day. I think that somewhere between 800-1200 pictures added is a good number, anything more than that and some good photos are missed by people browsing the database. If each of us was screening 50 shots per day, that would require 30 screeners, which is about where we are at. Of course, it isn't a 1:1 ratio as some pictures get screened by 2 or more screeners.

We are currently running at 7-9 days, I think we will see an improvement in the coming weeks, just in time for summer in the northern hemisphere   

I come up with another question...

A few posts ago it was said (if I got it correct) that screers do not look at the histogram and all that sort of things but they rely on their eyes.

now, I have received several rejections especially for dark and contrast having the histogram correct.

So, I'm wondering if, by pointing this out after a rejection (maybe in the appeal process), it would be profitable... I mean, would the screeners/headscreeners take a deeper look at the image to see if the histogram is really ok?
I know that this would take a lot of further time... but I think that, in some tricky cases, it would work...

Thanks in advance!  

Quoting teopilot (Reply 77): So, I'm wondering if, by pointing this out after a rejection (maybe in the appeal process), it would be profitable... I mean, would the screeners/headscreeners take a deeper look at the image to see if the histogram is really ok?

It wouldn't take that much time for a single image, but if we apply it to one, then to be fair, it should be applied to all, and that will not happen. In any case, you would find some people who would debate what is a 'correct' histogram. A histogram is just an aid to help determine the correct exposure, and shouldn't be a replacement for the human eye. It is sometimes quoted in rejections to help people see the problems with their image(s), but is certainly not the determining factor in a rejection/appeal.

Quoting teopilot (Reply 77): I have received several rejections especially for dark and contrast having the histogram correct

This serves to illustrate my earlier point. What we must consider when looking at a histogram is where in the image the dark and light pixels are, their nature and how they affect the overall exposure. Just because the histogram suggests there's an abnormal concentration of overly bright pixels doesn't lead to the conclusion that the image is over-exposed. For instance, if there is strong sunlight reflecting off something metallic in the background, that would lead to such a concentration; but that doesn't mean that the subject of the image isn't correctly exposed.

Exposure is something we try to get as accurate as possible, rather than something we can always get spot on. The laws of physics tell us that we can rarely be exact, but as long as we're somewhere not too far away we're winning.

It also depends what effect you want the photo to have. Sunset shots like those we tend to see here will be far from exact by the histogram, yet they are pleasing to the eye. They simply wouldn't look right if we processed them in accordance with the histogram.

Quoting dlowwa (Reply 78): what is a 'correct' histogram

I certainly wouldn't like to debate this one, as I'd say there's no true answer. Does a 'correct' histogram exist in reality? I guess a 'correct' histogram is one that goes with an image that looks right to the eye - and that puts us firmly back at square one!

Karl

Thanks Dana for the reply.  

Quoting dlowwa (Reply 78): It wouldn't take that much time for a single image, but if we apply it to one, then to be fair, it should be applied to all, and that will not happen.

And I think there's no point in objecting this... I was just wondering if it was applied to those tricky pictures, maybe as an help for the screening process.
Then of course, my question was also about the appeal, and you've hit both of them...
But I also understand that if you do like this with some pic, some others would have the same rights to be processed in the same "long and demanding" way.

Quoting dlowwa (Reply 78): shouldn't be a replacement for the human eye

Actually, to be honest, I didn't think about the "human factor" involved in the screening process: I mean those particular faculties that belong to men and that machines will never have. And the histogram is just a result of computing processes (so, do the math...)  
Probably I rely too much on it at times... LOL
Just because sometimes I need a precise reference for the editing: you know, when you aren't a well trained eye yet! But I hope to train my eye better... not only for aviation photography, but for photography in general!

Quoting dendrobatid (Reply 8): When we screen we can select batches of images, 1,5, 10 or 20 at which the images open and we see the thumbnails and information. The thumbnails give a good guide to the centre but we open every image and get a screen like this

I see nothing has changed since I was a screener from 1999-2001. Back then, AirNikon and I were screeners with a couple other folks (I think we pre-dated Gary, actually). He and I used to start long IM conversations about how we had page after page of terrible photos or how we'd get photos some clown took of his flight sim plane, or a painted piece of paper. We had some great nicknames for photographers (some of whom still post very well known photos to this day).

Oh, those were the days....

[Edited 2012-03-06 16:26:41]

Hi Chris

Long time no hear  

There was only one screener when I started and that was Big J!

Gary

Any responses to my questions re: Browsers, Screens, Calibration, Etc. above? thanks  

Quoting jaktrax (Reply 79): In any case, you would find some people who would debate what is a 'correct' histogram. A histogram is just an aid to help determine the correct exposure, and shouldn't be a replacement for the human eye

Dana, I agree and disagree at the same time. Definitely the human eye can tell when something looks right or wrong. But, you can also understand when it is wrong, meaning when the judgement is wrong, it can come back to haunt you. I say this because I have had a few rejections where the whites, midtones and shadows are exactly the same level as other shots of my own in the db as well as shots accepted around the same time. This is not to say the system should be changed but just a point to make that a lot of times, no exception is made for when it may seem wrong, but isn't

Perhaps if the rejection is appealed, a more stringent look into the rejection/appeal and perhaps it would be accepted when screened a bit more carefully. I know that I have even had some rejections for brightness where the difference has been so small as to not be noticeably better in the re-edit. Yes my opinion vs. yours but it should not be thirty or less vs. thousands  

Quoting je89_w (Reply 21): If only the personal message to photographer for an accepted photo could work again, that would make things easier for both the screeners and the photographers.

That would be great indeed! Screeners and photographers shouldn't only communicate with each other with rejections.
I'm only uploading fairly old slidescans, and the rules state that those are screened with more leeway than shots from the digital era. Of course that's a good thing, but I think I would welcome some feedback on accepted shots too. Maybe a shot used more "leeway" than needed, when the edit had been better in some way.

Now, I have the impression that screeners do communicate with the photographers in some ways. Like, every now and then I get a rejection on a shot I thought was "sure thing". Then I think that the shot was rejected, not because it wasn't good enough, but because it could be better with a bit more sharpening, or a bit more or less contrast. I don't mind such rejections, because I learn from them, but it would be better to see a remark like 'could do with a bit more sharpening' in accepted photo's.

And a point that I think is much better than it used to be: when a shot is rejected I know if it's worth to try again, or just give up. A few years ago, all my rejections had multiple rejection reasons. Some of those were impossible to do better, so I didn't know if fixing the fixable was worth the effort. Nowadays, the only rejection reasons I get are ones that I can work on. And if an impossible-to-fix rejection reason shows up, then I know I can just give up (or appeal).

Anyway, thumbs up for the screeners, you're doing a great job