Psych
Topic Author
Posts: 2944
Joined: Tue Nov 16, 2004 1:17 am

The Process Of Screening Debate

Thu Sep 14, 2006 9:10 pm

I have been a member of the site long enough now to recognise that there are periodic episodes of vocal disquiet about standards and how the site is run. We are in the middle of one of those now, prompted by Michael Carter's (Dacman) concurrent thread.

This is my attempt to start a process of rational debate about some of the central issues. My plea is that contributors keep the emotions out of the discussion, and there is some respect shown for opinions that might differ from our own. I make no apology if related issues have been discussed in the past - it can't hurt to give it a fresh look. If you are sick and tired of these kind of conversations, or have nothing useful to say, just move on to another thread of more interest to you.

Okay - the process of screening. Currently most photographs need three different screeners agreeing that the photo meets the site criteria before it is accepted. If two screeners pass the photo as acceptable, but one sees it as a rejection, the photo is rejected.

Some argue that this system works, is fair and there is no need to change it - one screener has found an element of the photo that doesn't meet the criteria, and that is enough. The protection against subjectivity or error is the appeals process. Others argue that it is fundamentally unfair and goes against the desire for consistency, as 2 to 1 should go in favour of the majority assessment, not the minority.

Let's have some debate about this. I would like to see people put forward the argument for things remaining as they are, and others the counter-argument. The various pro's and con's can then be thought through and possibly - just possibly - people may develop a better understanding of things.

I, for one, fully acknowledge that it is not in my gift to make any changes - the site has an owner and he has ultimate decision-making authority. It is not a democracy. But I believe part of this repeating cycle of distress amongst some photographers boils down to problems in communication - in its widest sense - and some misunderstandings about what really goes on.

Let's have a go at communicating.

Paul
 
linco22
Posts: 1279
Joined: Tue Jun 14, 2005 6:16 am

RE: The Process Of Screening Debate

Thu Sep 14, 2006 9:40 pm

Hi Paul,

How are you keeping? Thanks for getting something like this started. It would be good to keep all criticism constructive. With photography being a subjective thing the 'all 3 must agree' rule goes against this. I would like to see this changed.

The rejection emails are another big gripe for me. I have no problem with big ques etc etc, but when I get a rejection such as 'soft' or 'quality' I'd like to know more specifically what. Does this add a huge commitment on top of the screeners already? is this viable to put into place where each rejection, if neccessary, get a personal message. Say 'soft tail' 'too much noise in sky/fuselage/wherever'. I think this would help the rejection process alot.

Regards
Colin  

May I also add that this shouldnt be taken as screenr bashing. We all know their hard work and what they put up with. This should be a way to move things forward for the site I think

[Edited 2006-09-14 14:43:18]
 
timdegroot
Posts: 3258
Joined: Wed Apr 24, 2002 10:37 pm

RE: The Process Of Screening Debate

Thu Sep 14, 2006 10:04 pm

I'll make sure I'll reply to the suggestions made here as long as everyone "behaves". We've heard the complaints, we need solutions.

Tim
Alderman Exit
 
linco22
Posts: 1279
Joined: Tue Jun 14, 2005 6:16 am

RE: The Process Of Screening Debate

Thu Sep 14, 2006 10:08 pm

Quoting TimdeGroot (Reply 2):
I'll make sure I'll reply to the suggestions made here as long as everyone "behaves".

Couldn't agree more Tim, I hope it stays that way

Regards
Colin  Smile
 
Psych
Topic Author
Posts: 2944
Joined: Tue Nov 16, 2004 1:17 am

RE: The Process Of Screening Debate

Thu Sep 14, 2006 10:15 pm

Quoting Linco22 (Reply 1):
How are you keeping? Thanks for getting something like this started.

I'm okay Colin - keen to contribute to things getting better and ensure this is a good place to be. Thanks for your contribution so far.

Quoting TimdeGroot (Reply 2):
I'll make sure I'll reply to the suggestions made here

Great stuff Tim.

There are many aspects of the screening process about which many have strong feelings. For the purposes of clarity of discussion, can we focus on the one topic - i.e. 3 screeners saying 'yes' vs majority decision vs balloting vs 'blind' screening (i.e. the pros and cons of screeners knowing the previous views of colleagues) etc.

I would invite people to put forward what they would see as the optimum method for screening, and justify their reasoning. That way we can all consider the arguments - and hopefully understand better where different schools of thought are coming from.

Go ahead.....

Paul
 
timdegroot
Posts: 3258
Joined: Wed Apr 24, 2002 10:37 pm

RE: The Process Of Screening Debate

Thu Sep 14, 2006 10:18 pm

Please take into account take we can't have anything that will make the screening process more time consuming.

Tim
Alderman Exit
 
linco22
Posts: 1279
Joined: Tue Jun 14, 2005 6:16 am

RE: The Process Of Screening Debate

Thu Sep 14, 2006 10:22 pm

Quoting TimdeGroot (Reply 5):
anything that will make the screening process more time consuming.

I would assume that adding a personal to each rejection would be a prime example? For that particular agruement of mine.

Good to hear you're keeping well Paul

Regards
Colin  Smile
 
User avatar
ptrjong
Posts: 4109
Joined: Wed Mar 02, 2005 9:38 am

RE: The Process Of Screening Debate

Thu Sep 14, 2006 10:36 pm

Quoting Psych (Thread starter):
Some argue that this system works, is fair and there is no need to change it - one screener has found an element of the photo that doesn't meet the criteria, and that is enough. The protection against subjectivity or error is the appeals process. Others argue that it is fundamentally unfair and goes against the desire for consistency, as 2 to 1 should go in favour of the majority assessment, not the minority.

The 'ballot' system is interesting, but not without its problems. If everything was to be screened like this, a screener could not reject a hopeless cellphone photo on his own - two red lights would always be necessary. This would increase the screeners' workload in an unnecessary manner - after all, the majority of rejections are clear-cut cases or so we're told.

So I think a screener should still have the option of rejecting a picture and removing it from the queue when he's positive that it's unacceptable - just as he can direct add it when he's positive it's fine. However, when he doesn't like something about the photo but is not that positive, he should have the option to give it a 'red light' without removing it from the queue.

By the way, the 'direct add' option might be reconsidered. I think it's doing more bad than good, as it occassionaly leads to an enthousiastic acceptance of pictures which really should not have been accepted.

Peter Smile
The only difference between me and a madman is that I am not mad (Salvador Dali)
 
Woody001
Posts: 508
Joined: Mon Feb 23, 2004 7:39 am

RE: The Process Of Screening Debate

Thu Sep 14, 2006 10:39 pm

My opinion for what it's worth;

Firstly from an ex-screeners point of view, I can see the need for three screeners - I used to see things others didn't, obvious things like cropped borders, blurred images and dust spots. This wasn't a fault of the other two screeners, it was mainly down to me being a low volume screener and fresh eyes?
This also worked the other way around, other screeners would see things I missed.

I can confess to being on the side of not being that bothered to upload anymore images. I don't have that many images in the database <1K, but to have images rejected for stupid things like 0.02degs out of level is just driving photographers away.

About five photographers that I know well have said that they don't want to upload here anymore. One of them has just scanned in a whole batch of slides that his father took when he was a test pilot for Avro. He has recently had an image rejected for being unlevel - the image was taken out the window of his chase plane. He re-edited the image leveling it as best he could, re-uploaded it and it got rejected again for quality - it's a rare image that should have been given it's place on the site...

I'm going to stop now as this type of blanket rejection winds me up.

Ian.
If I could just get the afterburner working...
 
Stealthz
Posts: 5546
Joined: Wed Feb 02, 2005 11:43 am

RE: The Process Of Screening Debate

Thu Sep 14, 2006 10:58 pm

Quoting Ptrjong (Reply 7):
If everything was to be screened like this, a screener could not reject a hopeless cellphone photo on his own - two red lights would always be necessary

Not at all, a variation of the current "Instant add" could be used I would think the system could be setup so that "hopeless cellphone photos" or TV screen grabs could be "Instant rejected" but for images that made it past an initial culling a 2-1 vote could suffice.

I don't think it would be a huge stretch to allow a single screener the ability to reject an image like--
Big version: Width: 1024 Height: 683 File size: 72kb


Where as it might require a 2favour:1reject vote to accept an image like--
Big version: Width: 1024 Height: 683 File size: 164kb

(was a quick 30 second edit so likely fail even the 2:1 vote!!)

Cheers

[Edited 2006-09-14 16:17:28]
If your camera sends text messages, that could explain why your photos are rubbish!
 
JRadier
Posts: 3943
Joined: Mon Sep 27, 2004 11:36 pm

RE: The Process Of Screening Debate

Thu Sep 14, 2006 11:29 pm

Quoting Psych (Thread starter):
the site has an owner and he has ultimate decision-making authority.

Keep in mind that allthough this is Johans site, we, the photographers, have made it what it is and can reverse that as well. Thus we should have a saying in what goes on.

Quoting Ptrjong (Reply 7):
The 'ballot' system is interesting, but not without its problems. If everything was to be screened like this, a screener could not reject a hopeless cellphone photo on his own - two red lights would always be necessary. This would increase the screeners' workload in an unnecessary manner - after all, the majority of rejections are clear-cut cases or so we're told.

I agree with your ideas.

I see it all this way:

Screener 1: 3 choices, BAD (obviously not up to standards), NO (according to him there is something wrong, but some other screeners need to look at it) and YES.
Screener 2: get's both YES and NO from screener 1, votes as well. 2x NO is rejected, 2x yes is accepted, 1x NO, 1xYES goes to screener 3
Screener 3: Ultimate decision

Only drawback is that screener 1 can still veto a picture that might otherwise be accepted.
For once you have tasted flight you will walk the earth with your eyes turned skywards, for there you have been and ther
 
viv
Posts: 2953
Joined: Tue May 31, 2005 5:17 pm

RE: The Process Of Screening Debate

Thu Sep 14, 2006 11:31 pm

Quoting Woody001 (Reply 8):
a whole batch of slides that his father took when he was a test pilot for Avro

We need more of this type of shot, not less.

Conflict between "aviation-value" and photo-quality value"?
Nikon D700, Nikkor 80-400, Fuji X Pro 1, Fujinon 35 f/1.4, Fujinon 18 f/2
 
User avatar
ptrjong
Posts: 4109
Joined: Wed Mar 02, 2005 9:38 am

RE: The Process Of Screening Debate

Thu Sep 14, 2006 11:43 pm

Quoting JRadier (Reply 10):
Only drawback is that screener 1 can still veto a picture that might otherwise be accepted.

Yes, but I would think the screeners might be quite happy to exercise some self-restraint when not entirely sure, and use the NO vote option.

I don't think this system will increase screener workload. Some pics that today get rejected will be forwarded to a second or third screener, but on the other hand, pictures with two 'greens' don't require a third screening.

[Edited 2006-09-14 16:44:42]
The only difference between me and a madman is that I am not mad (Salvador Dali)
 
bubbles
Crew
Posts: 1124
Joined: Thu Apr 14, 2005 11:54 am

RE: The Process Of Screening Debate

Thu Sep 14, 2006 11:47 pm

Quoting JRadier (Reply 10):
Screener 1: 3 choices, BAD (obviously not up to standards), NO (according to him there is something wrong, but some other screeners need to look at it) and YES.
Screener 2: get's both YES and NO from screener 1, votes as well. 2x NO is rejected, 2x yes is accepted, 1x NO, 1xYES goes to screener 3
Screener 3: Ultimate decision

Hi Jurgen,

Just out of my curiosity ...

If the ultimate decision made by those three screeners is 2x No and 1x Yes, the photo is to be rejected, according to your example. But if the photographer decides to appeal that rejected image, and the Head Screener says "Yes", the decision will turn to be 2x No and 2x Yes. If so, what should the decision be - accept or reject?  Wink

_Hongyin_
 
linco22
Posts: 1279
Joined: Tue Jun 14, 2005 6:16 am

RE: The Process Of Screening Debate

Thu Sep 14, 2006 11:51 pm

Good point Hongyin. Perhaps the appeal decision remains different from the initial screening process. As it is anyway.

I hope some good comes out of this

Regards
Colin  Smile
 
User avatar
ptrjong
Posts: 4109
Joined: Wed Mar 02, 2005 9:38 am

RE: The Process Of Screening Debate

Fri Sep 15, 2006 12:03 am

Quoting Bubbles (Reply 13):

Sorry Hongyin, but it's not a good point. Think about it. If the appeal screener's say is not decisive, it's effectively impossible to accept any appeals at all Wink
But as Paul has proposed, let's stick to the basic screening procedure for now.
The only difference between me and a madman is that I am not mad (Salvador Dali)
 
Woody001
Posts: 508
Joined: Mon Feb 23, 2004 7:39 am

RE: The Process Of Screening Debate

Fri Sep 15, 2006 12:04 am

Quoting Viv (Reply 11):
Quoting Woody001 (Reply 8):
a whole batch of slides that his father took when he was a test pilot for Avro

We need more of this type of shot, not less.

Conflict between "aviation-value" and photo-quality value"?

Exactly!
Taken from the upload page:
We have lower rejection levels for rare shots (old, brand new, special paint scheme, accident, special situations etc).

Ian.
If I could just get the afterburner working...
 
bubbles
Crew
Posts: 1124
Joined: Thu Apr 14, 2005 11:54 am

RE: The Process Of Screening Debate

Fri Sep 15, 2006 12:14 am

Quoting Ptrjong (Reply 15):
But as Paul has proposed, let's stick to the basic screening procedure for now.

Sorry for my message which may sidetrack Paul's thread! Okay, I agree - let's continue discussing basic procedure for now.

_Hongyin_

[Edited 2006-09-14 17:16:27]
 
User avatar
acontador
Posts: 1381
Joined: Tue Jul 05, 2005 11:54 pm

RE: The Process Of Screening Debate

Fri Sep 15, 2006 12:53 am

Just my personal opinion for what it's worth:

1. Screening decision: For me, instant rejection/acceptance is the main problem, since the ultimate decision is left to one person only. I disagree that even for extremely bad shots this system would add CONSIDERABLY more work to the screeners, since these (obvious) bad shots will get screened very quickly. Therefore, the system "2 out of 3" seems like the best alternative, given the resource (screener time) constraint.
2. Rejection: If a picture is then rejected (2 NOs), then the message should be much more explicit than it is today. Particularly in the case of "quality" and "motiv", just stating these reasons alone do not help at all the photographer if he wants to improve and re upload that particular picture. I do not know exactly the screening system, but an idea could be to delete rejection option "quality" and include more subcategories in the others (or perhaps train screeners to include short personal messages more often).
3. Appeal: In order for this function to work properly, first of all the photographer needs to know EXACTLY why his picture was rejected in the first place (see 2.). I don't know exactly how many appeals the crew gets per day, but it would be also interesting to see the original screener and a head screener to discuss the appeal (and thus improved the learning curve of the screener), and again if rejected to be much more specific on reason.
4. Rules: Well, this site belongs to Johan, so he pretty much can set the rules he wants. However, in the particular case of motiv, the current ones are just not working for the community of photographers. A little more flexibility for the allowed motives and less subjective bad-motiv reasons would help a lot (a new thread could be started just for this one!).
Just sit back, relax and have a glass of Merlot...enjoy your life!
 
sulman
Posts: 1963
Joined: Wed Mar 03, 2004 5:09 am

RE: The Process Of Screening Debate

Fri Sep 15, 2006 1:06 am

The issue is the drive to technical quality over aesthetic quality.

That for me is the problem.
It takes a big man to admit they are wrong, and I am not a big man.
 
timdegroot
Posts: 3258
Joined: Wed Apr 24, 2002 10:37 pm

RE: The Process Of Screening Debate

Fri Sep 15, 2006 2:44 am

Ok here we go. I consider all issues above closed after I replied, otherwise there'll be no end.

Quoting Ptrjong (Reply 7):
By the way, the 'direct add' option might be reconsidered. I think it's doing more bad than good, as it occassionaly leads to an enthousiastic acceptance of pictures which really should not have been accepted.

It does more good than bad. It speeds up the screening process tremendously. If gross mistakes are made photos are usually deleted.

However:

Quoting Woody001 (Reply 8):
Firstly from an ex-screeners point of view, I can see the need for three screeners

When screening hundreds of shots a day as some screeners do, it can be too much to ask look at each images for minutes to ensure a direct add. The 3 screener system works well as Woody001 explains. Some screeners like to screen 10 images and instant add them all, taking 15 minutes, some like to do 30 in those 15 minutes and just HQ.

Quoting Woody001 (Reply 8):
but to have images rejected for stupid things like 0.02degs out of level is just driving photographers away.

This should not happen. If so, appeal. I know you might generalize here, but if shots get rejected for very minor things consider appealing. Having said that, if we spot and off angle image that is otherwise fine we reject because it's easy to fix. Therefore it might appear we reject for 'stupid' reasons, but we just want a better version.

Quoting JRadier (Reply 10):
Keep in mind that allthough this is Johans site, we, the photographers, have made it what it is and can reverse that as well. Thus we should have a saying in what goes on.

You do now don't you. Still, this is not a democracy. Also remember that many photogs are perfectly happy with A.net.

Quoting Viv (Reply 11):
We need more of this type of shot, not less.

Conflict between "aviation-value" and photo-quality value"?

Show me these rejections please.

Quoting Sulman (Reply 19):
The issue is the drive to technical quality over aesthetic quality.

It has been like that for a loooong time now. It's the basic motto of the site.

I appreciate you all thinking about a new screening process, but without inside knowledge of how the system works now you don't do the current system justice. We have the option to add comments, and second opinion and use this frequently. A rejection is thus not necessarily the product of one screeners but can be the product of 2 screeners who added comments, a second opinion with comments and a final reject.

Quoting Acontador (Reply 18):
4. Rules: Well, this site belongs to Johan, so he pretty much can set the rules he wants. However, in the particular case of motiv, the current ones are just not working for the community of photographers. A little more flexibility for the allowed motives and less subjective bad-motiv reasons would help a lot (a new thread could be started just for this one!).

I undertstand that the motive/centering issue needs work. We'll try to figure something out to make it clear what we want and don't want.

Quoting Bubbles (Reply 13):
Screener 1: 3 choices, BAD (obviously not up to standards), NO (according to him there is something wrong, but some other screeners need to look at it) and YES.
Screener 2: get's both YES and NO from screener 1, votes as well. 2x NO is rejected, 2x yes is accepted, 1x NO, 1xYES goes to screener 3
Screener 3: Ultimate decision

This would lengthen the screening process too much.

Decisions need to be made more quickly rather than more slowly. The key is to have a screening team that is on one line. This will take more work. We realize the problem but we want to solve it using a quality approach.

Tim
Alderman Exit
 
timdegroot
Posts: 3258
Joined: Wed Apr 24, 2002 10:37 pm

RE: The Process Of Screening Debate

Fri Sep 15, 2006 3:18 am

Keep this on topic. It's quite clear the photographers concerns are listened to and adressed. But we will never be forced to make change if 50 photographers on the forum say it must be so.

Tim
Alderman Exit
 
Psych
Topic Author
Posts: 2944
Joined: Tue Nov 16, 2004 1:17 am

RE: The Process Of Screening Debate

Fri Sep 15, 2006 3:20 am

Quoting Psych (Thread starter):
My plea is that contributors keep the emotions out of the discussion, and there is some respect shown for opinions that might differ from our own.

Nik..... banghead 

Something just comes to my mind - I can see the appeal of the 'majority vote' screening system, because it feels fair, but I also understand Ian's point, above, that one screener may spot something that others have missed.

However, I think there is a distinction here that warrants further thought. I think there is a qualitative difference between screener number 3 spotting something 'concrete', like a bad border/dirt/level etc, that others may have missed, and that person making a subjective judgement that is at odds with the judgement of two colleagues.

To illustrate further:

Example 1 - My photo is HQ'd by two screeners, both of whom were screening a number of shots and thus did not spend ages on the one image. It looked pretty good to them. The third screener knows that the ultimate decision is theirs and so they are looking very closely, and they see a dust spot. The photographer can reupload a cleaned up version, which maintains the high standards of the site, so they reject. Yes, that was 2:1, but there is some clear reason.

Example 2 - My photo is HQ'd by two screeners, who like my motive (say, a close crop). Screener three sees it and has strong views about a particular motive, because their interpretation of the criterion is that the photo falls foul of it. They reject the photo.

Example 1 was rejected on the basis of an objective flaw; Example 2 a subjective one. Now of course there are many of the A.net criteria that are subjective - but when the rejection is for these reasons, then is the 'majority' decision not the more equitable? Personally I am thinking particularly about 'quality' and 'motive', but there will be others.

There seems something reasonable about this position. However, life is seldom like that. I would also accept that it is unreasonable to have a system that requires all photos to be rejected by at least two screeners. That would mean that the kind of examples people quote - like the mobile phone shots - would have to be passed on even though it is glaringly obvious to most that it is an instant reject. The problem is that the baby gets thrown out with the bath-water. Maybe only photos that are obviously no-no's should be instantly rejected - however that is defined! Everything else goes forward for a second look. Would this take up a lot more time? I will defer to those who have more inside knowledge of the system to answer that one.

When the rejection is for a subjective decision, should that be confirmed by at least two screeners? Happy to hear counter-arguments.

Cheers.

Paul
 
User avatar
acontador
Posts: 1381
Joined: Tue Jul 05, 2005 11:54 pm

RE: The Process Of Screening Debate

Fri Sep 15, 2006 3:23 am

Hi Tim,

Quoting TimdeGroot (Reply 20):
Decisions need to be made more quickly rather than more slowly.

Really? Why? Because that's what the boss said???  Wink
I'd rather wait a couple of days longer for my pictures to be screened than waisting so much time trying to find out what was wrong with them in the first place. There is such a fixation with speed (probably from the days of a 14K queue?) but much less on quality and service (to the photog).

You answered on 4. but left 1. to 3. out my friend...

In any case, as I understand this is not just "bitching on screeners for free" but rather discussing and bringing forward ideas for improving the site, isn't it?
Just sit back, relax and have a glass of Merlot...enjoy your life!
 
timdegroot
Posts: 3258
Joined: Wed Apr 24, 2002 10:37 pm

RE: The Process Of Screening Debate

Fri Sep 15, 2006 3:26 am

Motive can be subjective too.

I think you'd find it a rare case when photos have 2 hqs and are rejected instantly for motive by the third screener. A second opinion would be much more common.

Paul your idea would meet with the same problem. What is a clear no-no. For us screeners an unlevel image is a clear no as well.

Anyway, you might just think I'm talking out of my ass since you have no way of ever knowing. But trust me we don't think lightly of screening, but we do have to make quick decisions, otherwise there'd be threads about the queue being at 15k again.


Tim
Alderman Exit
 
codeshare
Posts: 1689
Joined: Tue Sep 10, 2002 2:23 am

RE: The Process Of Screening Debate

Fri Sep 15, 2006 3:41 am

How about somehow splitting the process into first: motiv-wise reasons (ie. suitable for the DB) and then quality-wise reasons, so that, like Psych stated in the examples, situations could be avoided ? Posiibly, but could be more time consuming for the whole process.

KS/codeshare
How much A is there is Airliners Net ? 0 or nothing ?
 
INNflight
Posts: 3526
Joined: Tue Apr 13, 2004 5:11 am

RE: The Process Of Screening Debate

Fri Sep 15, 2006 3:41 am

Quoting TimdeGroot (Reply 20):
I understand that the motive/centering issue needs work. We'll try to figure something out to make it clear what we want and don't want.

Can't tell you how grateful I am to see this written here Tim. I'm a 100% sure in the end a more open policy to approach centered, (distance?) and motive photos will attract more photographers (who care less about reg's and a/c types) but shoot for the passion of getting the best light, powerful compositions, etc... I'm sure my point gets across.  Smile

In the end I assume the site would benefit. Of course no one asks you guys to accept a shot out of center or too far away. Motivational would be the key here.

To add my 2c to the screening process (the screening emails to be exact):

I do have the feeling personal messages of screeners should either get a better (more at the top) position in the email, or highlighted somehow, either in color or being bold.

I just experience this from time to time that scrolling down the email and picking out the couple of words from a screener isn't always easy. Maybe an own box below the rejection reason would make it easier, especially for inexperienced uploaders.

In the end, I have to admit I'm very happy at the moment with the way screening is done, and quite a few photos that were bordercase for me got accepted surprisingly. Thanks to you all.

Take care,
F.
Jet Visuals
 
Psych
Topic Author
Posts: 2944
Joined: Tue Nov 16, 2004 1:17 am

RE: The Process Of Screening Debate

Fri Sep 15, 2006 3:49 am

Sorry Tim - I am not quite clear which bits your comments are relating to there.

I agree motive can often be subjective, and I agree defining what counts as a 'no-no' could be a bit of a nightmare.

I suppose my thinking aloud - or should it be thinking allowed  wink  - is about refining the system such that when a photographer receives a rejection for a 'subjective' reason - i.e. one that some might agree with, others might not - then they know that at least two screeners will have seen it that way - not just one. They may not like it - and will have the facility of an appeal - but at least they can be confident that it is not the opinion of just one.

Paul
 
IL76
Posts: 2237
Joined: Mon Jan 05, 2004 5:43 am

RE: The Process Of Screening Debate

Fri Sep 15, 2006 3:55 am

Motive is subjective and mainly comes down to: "Does it look right?". If the light, the chosen angle, centering, the plane type (f.e. an F16 could work with a certain crop/angle, a Cessna 152 not) and surroundings work and it's clear what the photographer intended to show, there is no motive problem.
Just too many cases are the "This just doesn't look right"-kind. If some factor I mentioned above is missing/disturbing, the odds on rejection increase.
Yes, this often comes down to the individual screeners taste... But difficult cases almost always go through multiple screeners.

E
 
bubbles
Crew
Posts: 1124
Joined: Thu Apr 14, 2005 11:54 am

RE: The Process Of Screening Debate

Fri Sep 15, 2006 4:06 am

Quoting TimdeGroot (Reply 20):
This would lengthen the screening process too much.

Hi Tim,

That is just what I quoted from Jurgen in Reply 10.  Wink

_Hongyin_
 
Avro85
Posts: 236
Joined: Sat Jun 11, 2005 4:35 am

RE: The Process Of Screening Debate

Fri Sep 15, 2006 4:12 am

First of all I'd like to thank Paul for starting this very interesting and constructive topic.

Secondly I didn't have the time to read all answer so don't shoot me if my comment is the same as someone else made...  Wink

Quoting TimdeGroot (Reply 20):
I undertstand that the motive/centering issue needs work. We'll try to figure something out to make it clear what we want and don't want.

I also agree that the centering and motiv can be a very subjective matter, however I think the problem could be ("partially") solved. Why not ask people who upload a picture with intentionally unusual centering or motiv to write a motivation in the "comment to screener box". Herewith the uploader will be able to give a short explanation to the screeners WHY he/she made the centering/motiv like it is. This would then help the screeners to interpret and understand the motivation behind the picture.

This would not require more work to the screener (reading 2-3 lines of text takes a few seconds  Wink ) and would certainly help to make a decision on the motiv or centering issues.

Just my 2 cents

Chris
 
sulman
Posts: 1963
Joined: Wed Mar 03, 2004 5:09 am

RE: The Process Of Screening Debate

Fri Sep 15, 2006 5:20 am

Quoting TimdeGroot (Reply 20):
It has been like that for a loooong time now. It's the basic motto of the site.

Tim,

It's very interesting - to me - that you state that like it's a universal truth that is not questioned, because it is this very ethos that has taken the site to this point of contention.

Firstly, it's an extremely subjective 'thing' to idealise. How many contributors - if they're really honest - have looked at some other people's rejections and just thought "Oh for Christ's sake, that's just silly." Likewise how many rejections (a recent sharpening one springs to mind, discussed on on this forum) have people - even a screener or two - clutching at proverbial straws because basically nobody has a clue what's wrong with the rejected image.

The site is at an interesting point at the moment. I think technical quality standards have been driven to such a high level that they are misunderstood; and I would charge the screeners with that as much the contributors - nobody really knows what they are doing wrong or right. This really needs to change.


James
It takes a big man to admit they are wrong, and I am not a big man.
 
eadster
Posts: 2125
Joined: Tue Jan 25, 2005 7:31 pm

RE: The Process Of Screening Debate

Fri Sep 15, 2006 5:23 am

How are we going to have a nice communication disscussion when NIK clearly makes matters worse?? Always drags communications through the mud.

Anyway...

This may have been dissussed before, but my main problem with screening at the moment is multiple rejection reasons over mutiple upload. We all know that screeners see many things that the site does not want that us photographers don't.

So I upload a shot, it gets booted for level, I level it, then booted again for soft, fix that, the booted for motive. See what I'm getting at? Some would be banned for doing that I suppose so is there a problem with ticking all boxes to say level, soft, motive. Then I know that even though I fix the level, the soft that I need not bother because the motive is not what anet wants.

I feel that if an image is rejected for all reasons all at once it again is going to help us by 1. Even smaller Q (which is really good currently, well done to all involved) and 2. will help us and especially new comers to the site who have basically no upload slots to begin with. They then can work more closely with their editing and get closer to their first acceptance.

Secondly I'd like to mention the screening itself. I don't know if this is true. but I hear that a shot is screened by 3 screeners. Screener A and B like it, screener C is not too fond of it so it goes. Is that right?

If so, if A and B think its ok then surely C can be overwritten as clearly there are many good points about the shot. If C finds something that can be easily fixed (baring in mind A and B did not see this small error), then a short note to the Photographer saying what the issue is, upload again when fixed.

Lastly - Night/Dusk/Dawn

A While back I had a lot of trouble with some shots taken on dusk with very low sun showing some great colours. I uploaded as night/dusk/dawn. Only to be told that they clearly showed sunlight on them. Having checked the relevant section off one of the front pages of the site, revealed shots with sunlight in them. To this day I have not received an answer to this issue. Some shots of mine were rejected for colour, due to the sun's angle, but once ticked night/dusk/dawn they were in! I just need some clarification on this.

Hope you all understand what I'm getting at and would be glad to disscuss further.

Martin
 
Psych
Topic Author
Posts: 2944
Joined: Tue Nov 16, 2004 1:17 am

RE: The Process Of Screening Debate

Fri Sep 15, 2006 5:36 am

Quoting Psych (Thread starter):
My plea is that contributors keep the emotions out of the discussion, and there is some respect shown for opinions that might differ from our own.

Even though I am 44, and have been a member here for 2 years, I still have this childlike, naive notion that the above should be possible  sigh .
 
javibi
Posts: 1295
Joined: Sat Oct 02, 2004 5:55 pm

RE: The Process Of Screening Debate

Fri Sep 15, 2006 6:35 am

Give the screeners five choices for every picture:
-2,-1,0,1,2
If a picture sums up 2 points it gets added, if the sum reaches -2 is rejected. If the screener marks -2 or -1 he has to mark the reason for rejection as well.
That way we have instant add (+2), instant reject (-2), second opinion (0), HQ (1) and "IMO it is a reject but let's see what the other guys think" (-1).

The total screening time shouldn't be longer, IMHO. Any opinions?

Regards.

j

P.S: There is always the possibility of looking at the username at just not be reading his posts, I am sure that could help to keep the thread on track  Wink
 
timdegroot
Posts: 3258
Joined: Wed Apr 24, 2002 10:37 pm

RE: The Process Of Screening Debate

Fri Sep 15, 2006 6:37 am

Quoting Javibi (Reply 41):
The total screening time shouldn't be longer, IMHO. Any opinions?

Correct me if I'm wrong but it's basically the same thing we have now? Only the -1 we don't have but it can be a hq with comments just as well.

Tim
Alderman Exit
 
javibi
Posts: 1295
Joined: Sat Oct 02, 2004 5:55 pm

RE: The Process Of Screening Debate

Fri Sep 15, 2006 6:44 am

Quoting TimdeGroot (Reply 42):
Correct me if I'm wrong but it's basically the same thing we have now?

No, Tim. 2 HQ (1+1) will mean direct add; negative points (opinions) can be cancelled by positive points. Not the same at all, IMHO.

j
 
timdegroot
Posts: 3258
Joined: Wed Apr 24, 2002 10:37 pm

RE: The Process Of Screening Debate

Fri Sep 15, 2006 6:57 am

Quoting Javibi (Reply 43):
2 HQ (1+1) will mean direct add

True but we're already doing this a lot. But if the negative points need to be cancelled it would take longer?

Tim
Alderman Exit
 
linco22
Posts: 1279
Joined: Tue Jun 14, 2005 6:16 am

RE: The Process Of Screening Debate

Fri Sep 15, 2006 7:08 am

Maybe the current system works fine??? Maybe their isn't a clearer way of doing things. I think concentration, for me, should be made clear as to what is required. But I havent been in great form with and without the camera lately. Maybe we over analyse all this?

Claerly we have a site here with the best aviation collection online. We are losing top notch photographers due to clarification of rules and how they are implemented. Are we going the wrong path with changing the screening process? Will that solve the real problem. Inconsistancy of rules not only by us but by the screenign team. We're all human an sometimes it doesn't quite work out the way it should.

I'm confident we can resolve these issues.

Regards
Colin  Smile
 
javibi
Posts: 1295
Joined: Sat Oct 02, 2004 5:55 pm

RE: The Process Of Screening Debate

Fri Sep 15, 2006 7:12 am

Quoting TimdeGroot (Reply 37):
But if the negative points need to be cancelled it would take longer?

Bordercases would take long, as I think they take now, so no changes here. But my system reduces the chances that a single screener rejects a picture against the opinion of other screeners.

E.g:
1) Total crap: 1st screener rates -2, rejected (has to state reason as always)
2) Picture with some flaws: 1st screener rates -1, 2nd rates -1, gets rejected (both have to state their reasons, as they are rating minus points)
3) Another picture: 1st screener thinks it is ok, so +1. 2nd agrees: +1. Total 2, in it is!
4) Another: 1st screener thinks it is ok, so +1. 2nd sees a problem: -1. 3rd agrees with 2nd: -1 (or -2, in which case it'll be rejected). If he chooses -1, total is now -1, so we need another screener...

See my point? Only bordercases would take long, but they do already, I think. Good pictures can actually get added with shorter in screening time.

Think about it, please!  Smile

j
 
User avatar
ThierryD
Crew
Posts: 2029
Joined: Sun Dec 04, 2005 6:58 pm

RE: The Process Of Screening Debate

Fri Sep 15, 2006 7:44 am

Quoting Javibi (Reply 39):

Nice idea there by Javier however I think the screening process should be kept easier.
How about that (and sorry if I oversee something but I just tried to get outside the problem to find a simple solution):
based on the votes of 3 screeners who'll have 3 options (negative, neutral, positive):
2 neg --> reject
2 pos --> accept
neutral --> over to headscreener for final decision
This would eliminate instant rejects or accepts but photos qualifying for any of those would get a quick immediate rej. or add. by 2 screeners also and would avoid subjective decisions of only 1 screener.
On the other hand the simplicity of the system would probably more than compensate the time it takes for at least 2 screeners to check any uploaded photo.

For a 2 neg rejection the standard rejection reasons would be sent as they should be quite clear to see and in case of a headscreener reject a short personal message as to the exact reason would be added.

Thierry
"Go ahead...make my day"
 
timdegroot
Posts: 3258
Joined: Wed Apr 24, 2002 10:37 pm

RE: The Process Of Screening Debate

Fri Sep 15, 2006 7:47 am

Believe me if we would get rid of instant adds and especially rejects the queue would skyrocket.

Tim
Alderman Exit
 
User avatar
ThierryD
Crew
Posts: 2029
Joined: Sun Dec 04, 2005 6:58 pm

RE: The Process Of Screening Debate

Fri Sep 15, 2006 7:54 am

Quoting TimdeGroot (Reply 41):
Believe me if we would get rid of instant adds and especially rejects the queue would skyrocket.

Ok, than just for my (and maybe other people's) understanding, on what criteria do you base an instant accept (reject)? And what is the time a typical instant add (rej.) would take?
For instance I had an F-15 shot at Leeuwarden instantly accepted while other shots whose (technical) quality I believe to be higher went the normal (longer) way.

Thierry
"Go ahead...make my day"
 
timdegroot
Posts: 3258
Joined: Wed Apr 24, 2002 10:37 pm

RE: The Process Of Screening Debate

Fri Sep 15, 2006 7:58 am

Quoting ThierryD (Reply 42):
Ok, than just for my (and maybe other people's) understanding, on what criteria do you base an instant accept (reject)? And what is the time a typical instant add (rej.) would take?

Criteria are the same. Screening can take 20 second right up to several minutes.

Thierry, your images might have been screened by different screeners. As said not everyone used instant adds as much.

Tim
Alderman Exit
 
User avatar
ThierryD
Crew
Posts: 2029
Joined: Sun Dec 04, 2005 6:58 pm

RE: The Process Of Screening Debate

Fri Sep 15, 2006 8:09 am

Well then if the proposed screening system was in place a single screener could spend even less time for an instant add/rej as he knows that anyway a second screener would have a look and if both (don't) see a major problem the picture would be (added) rejected so instead of having 1 screener spend lets say 2 minutes on a shot you'd have 2 screeners spending 1 min thus time would be the same but the objectivity of the screening would improve!?

Thierry
"Go ahead...make my day"
 
Psych
Topic Author
Posts: 2944
Joined: Tue Nov 16, 2004 1:17 am

RE: The Process Of Screening Debate

Fri Sep 15, 2006 5:40 pm

My thanks to the moderator(s) who have cleaned this thread up and just left readers to look at the substance of the debate.

I hope all will perceive this as an opportunity to voice constructive views about the current state of play and make suggestions about how things could - in theory - be different, and what the potential effects of any changes would be - both good and bad. Also it remains an opportunity to debate with others why things are best left as they are now, if that is your point of view.

My thanks to all who have contributed so far - it now looks like a discussion as it should be  thumbsup .

All the best.

Paul
 
timdegroot
Posts: 3258
Joined: Wed Apr 24, 2002 10:37 pm

RE: The Process Of Screening Debate

Fri Sep 15, 2006 5:53 pm

Quoting ThierryD (Reply 44):
rejected so instead of having 1 screener spend lets say 2 minutes on a shot you'd have 2 screeners spending 1 min thus time would be the same but the objectivity of the screening would improve!?

In theory yes, but this is not how it works in practice. They would both spend 2 minutes, because at A.net we screen every image like it was screened the first time, we don't assume screener X saw all the dustspots or corrected some info. Screening can't really be broken down into math, there are too many variables, the screeners themselves being the most important one.

Let me give some real world examples of how shots get screened in different ways, showing that we don't use just a simple 3 HQ or instant reject system

Added after 2 HQs
Priority add after 1 HQ TristarYUL Michael Durning 2006-09-15 03:39:56 AirNikon rates HQ: TristarYUL Priority Instant Adds:

Added after 1 HQs with one SO, so this could have been an instant reject but was in fact later added by two screeners.
Priority add after 1 HQ Riley Bjorn van der Velpen 2006-09-15 02:14:25 TimdeGroot asks for second opinion: skyAirNikon asks for second opinion: TimdeGroot asks for second opinion: skyAirNikon rates HQ: TristarYUL rates HQ: Riley Priority Instant Adds:

Added after 3 HQs, but note the comment on the second. Again we don't just instantly reject.
Added after 3 HQ Riley Jan Severijns - AirTeamImages 2006-09-15 02:14:25 AirNikon rates HQ: TristarYUL rates HQ: slight angleRiley rates HQ:

I can't show the same for the rejections, but you will see a variety of combinations used here. I can't break our process down in numbers like Javier, but it does often work in a similar way. Also remember that lots and lots of rejections and acceptions are very clear cut and really don't need 5 screeners to look at them. It's when we look at "creative" stuff or bordercases when you'll see many of these varieties used.

Tim
Alderman Exit
 
PUnmuth@VIE
Posts: 3119
Joined: Thu Aug 24, 2000 9:31 pm

RE: The Process Of Screening Debate

Fri Sep 15, 2006 6:13 pm

I think the current screening system works pretty well as shown by the examples Tim brought up. But its just the system that works good if used properly. I think the main issue with the process is that it is very hard to maintain consistency across the 23 screeners engaged in the process.
I remember that there where many discussions going on behind the scenes about various screening issues and i assume its the same if not more today. As long as not all of the 23 are willing to participate in those team internal discussions or at least follow the results of those discussions (not to mention not to ignore the results on purpose) the debates here in the photo forum will appear over and over again.
Maybe there have been steps taken to achieve this target but from the debates here and the not understandable rejections and sometimes also unwarranted additions to the database it still looks there is much room left for improvement inside the screening crew itself.
Main issues seem to be
- motive
- priority
The other issue is the culture here in the forum itself. As long as some are only here to post nonconstructive submissions we can't blame the screeners not to come here but that's worth a thread on its own.
And I think Tim deserves great respect for staying in the discussion as
the only headscreener.
-
 
dendrobatid
Posts: 1645
Joined: Sat Nov 06, 2004 3:40 pm

RE: The Process Of Screening Debate

Fri Sep 15, 2006 6:16 pm

quote=JRadier,reply=10]I see it all this way:

Screener 1: 3 choices, BAD (obviously not up to standards), NO (according to him there is something wrong, but some other screeners need to look at it) and YES.
Screener 2: get's both YES and NO from screener 1, votes as well. 2x NO is rejected, 2x yes is accepted, 1x NO, 1xYES goes to screener 3
Screener 3: Ultimate decision
Only drawback is that screener 1 can still veto a picture that might otherwise be accepted.[/quote]

This sounds a quite remarkable idea, in fact almost exactly what happens at present ! One person can however reject just as one person can add, but that is the exception rather than the rule I have not been screening long and am not deeply ingrained with it and just a few weeks ago I shared much of the scepticism that is being shown in this and the other thread. My view has changed considerably since I have now seen this side of things. Any feelings I had of unfairness have gone, not just dimmed, totally gone. Consistency is of course a problem as we are making subjective decisions and no two people will agree all of the time. I made a mistake the other evening and accepted one that I should not have but was contacted by a Head as a sort of quality control. The image remains as it would be unfair on the contributor and is now a beacon to the fallability of screeners. Of course things are not perfect, subjective decisions are going to vary but the system does actually work pretty well most of the time. A bit of fine-tuning might not go amiss but I have found the Headscreeners receptive to new ideas (as demonstrated by Tim here)

There is another point, perhaps not considered by contributors. There is a small team of screeners of which I am proud to be a part, but that small team is subject to the scrutiny of thousands of pairs of eyes on a daily basis. Surely that is a recipe for disagreement ?

At least the gripe about long queues has largely gone !

Quoting Linco22 (Reply 6):would assume that adding a personal to each rejection would be a prime example? For that particular argument of mine.[/quote] (for increasing screening time)

I think that most of us try when a rejection is contentious or not obvious but they are not always necessary, dark, soft, oversharp, double etc. The problem one is quality when a lot of minor things can all add up to fail an image. I used to get plenty of personals, don't you ?

Quoting Woody001 (Reply 8):Firstly from an ex-screeners point of view, I can see the need for three screeners - I used to see things others didn't, obvious things like cropped borders, blurred images and dust spots. This wasn't a fault of the other two screeners, it was mainly down to me being a low volume screener and fresh eyes? This also worked the other way around, other screeners would see things I missed.[/quote]

Woody did this for a lot longer than I have and sums it up perfectly there. Those like Woody, like Tim, like myself, all of whom know the system, are all arguing that not much change is needed

Mick Bajcar
 
beechcraft
Posts: 731
Joined: Sun Nov 16, 2003 2:10 am

RE: The Process Of Screening Debate

Fri Sep 15, 2006 6:19 pm

Hi all,

Quoting Psych (Reply 22):
Example 2 - My photo is HQ'd by two screeners, who like my motive (say, a close crop). Screener three sees it and has strong views about a particular motive, because their interpretation of the criterion is that the photo falls foul of it. They reject the photo.



Quoting TimdeGroot (Reply 24):
I think you'd find it a rare case when photos have 2 hqs and are rejected instantly for motive by the third screener. A second opinion would be much more common.

I think the vast majority of screeners is doing just that. Often enough there are a lot more than three screeners involved in the whole process through the "second opinion" option.

I´m a bit short on time right now, so i´ll just post a quick reply....

Overall i think the screening process i working quite fine. There are are some issues that need to be worked on, as mentioned before, basically the motive/centered and quality rejections, i guess.

But the three screener accept/ one screener rejects system itself is working good, regarding the big picture.

The main problem of the whole process is that it involves human beings, which tend to make mistakes. Also, the screening team is not a homogenic mass, but consists of a lot of people with different grades of experience. New screeners are coming in from time to time, while some old ones are leaving, etc.

Is think one of the main goals would be to a have an expierenced medium size screening team, rather than a very big one.

I urge people to add more personal notes to the screeners as wll as the screener should try to send more personal remarks.

Also, if you feel your photo was rejected wrongly, appeal it.

Gotta catch a plane, sorry if that was a bit confuse,

take care,

Denis
That's it! You people have stood in my way long enough. I'm going to clown college!

Who is online

Users browsing this forum: No registered users and 6 guests