Sponsor Message:
Aviation Photography Forum
My Starred Topics | Profile | New Topic | Forum Index | Help | Search 
Small Change To How Photos Are Added To The Db  
User currently offlineAdministrator From Sweden, joined May 1999, 3251 posts.
Posted (8 years 2 months 2 weeks 20 hours ago) and read 5076 times:
AIRLINERS.NET CREW
SITE ADMIN

Hi guys,

First let me start by apologizing to the photographers that had photos lost during the weekend, about 300 entries had to be removed due to database problems. It should all be sorted out now and the effected photographers have been notified to upload again with priority screening. We have taken measures to see to that this does not happen again.

I would like to tell you about a small change to the screening process. Photos that are accepted by the screeners are not added immediately to the database but are queued and added in bulk when there's 10 or more shots waiting to be added. This is why some of you have reported photos as "lost" when you cannot find them on the photo stats page nor in the database. I have added info about this queue to both upload stats pages under MyPhotos. Should any of your photos be queued for addition to the database, it will be mentioned on the stats pages.

This change serves to purposes:
1) Simplify the screening script. During high load, the screening process can take a very long time as the script has to insert the new entries one by one, copy all relevant photo files to all three servers (one main + two backup) and send out emails to photographers. This has sometimes caused timeouts and other problems. By moving the inserts into the database, the copying of photo files and the emailing of photographers to a server based program, I am hoping the situation will improve dramatically.
2) By collecting at least 10 photos before making an insert into the main database, we save precious resources as we can make one big insert with all photos waiting instead of doing 10 or more individual inserts. This has very positive effects on server load and query cache.

I could also mention another small change I did recently - extended the photo ID number to include seven digits instead of six. A small change it might seem but there's a heck of a lot of work behind it. All scripts and databases on the site that deal with photos (there's a lot of those!) had to be checked and some re-written to handle the longer ID field. As you can see, all photo ID numbers now start with a zero but that will not last long. As of writing this, we are 700 photos away from the magic ID of 1,000,000. This is naturally why this change had to be made. If I had not, we would not be able to move beyond ID 999,999.
Note that this change does not effect links to photos. They can be linked to with or without the leading zero. For instance, http://www.airliners.net/open.file?id=0999111 and http://www.airliners.net/open.file?id=999111 works equally well.

Finally, I am happy to say that the new server is now installed and in a semi-test production phase. It is currently serving photos under photos.airliners.net and doing some SQL tasks. I will continue to test it for speed and stability for a few more weeks before moving it to its intended role as the main webserver.

Thank you for showing your work on Airliners.net - you're what makes the site what it is!

Best Regards,
Johan


Working on the site from morning 'till night that's livin' alright (1997-2007)
8 replies: All unread, jump to last
 
User currently offlineAndrewUber From United States of America, joined Jul 2003, 2528 posts, RR: 41
Reply 1, posted (8 years 2 months 2 weeks 20 hours ago) and read 5074 times:

Thanks for the update! Glad to see almost a million photos on airliners.net!!

Drew



I'd rather shoot BAD_MOTIVE
User currently offlineGary2880 From , joined Dec 1969, posts, RR:
Reply 2, posted (8 years 2 months 2 weeks 20 hours ago) and read 5057 times:

so.... we have to get 10 photos accepted before they appear on the database..

User currently offlineViv From Ireland, joined May 2005, 3142 posts, RR: 29
Reply 3, posted (8 years 2 months 2 weeks 20 hours ago) and read 5057 times:

It still takes about 9 days to get a shot accepted.


Nikon D700, Nikkor 80-400, Fuji X Pro 1, Fujinon 35 f/1.4, Fujinon 18 f/2
User currently offlineIL76 From Netherlands, joined Jan 2004, 2237 posts, RR: 50
Reply 4, posted (8 years 2 months 2 weeks 20 hours ago) and read 5049 times:

Quoting Gary2880 (Reply 2):
so.... we have to get 10 photos accepted before they appear on the database..

No, 10 photos (not just yours) need to have been accepted. If I screen a set of 10 photos, and as a result 5 photos have been rated HQ 3 times, these 5 will be added when at least 5 others have also been accepted.
So additions are done in batches of at least 10 photos to reduce the load on the servers. It's a technical thing, it has nothing to do with the screening process.

[Edited 2006-02-06 11:49:08]

User currently offlineRotate From Switzerland, joined Feb 2003, 1488 posts, RR: 16
Reply 5, posted (8 years 2 months 2 weeks 19 hours ago) and read 5025 times:
Support Airliners.net - become a First Class Member!

Quoting Administrator (Thread starter):
10 or more shots waiting to be added



Quoting IL76 (Reply 4):
So additions are done in batches of at least 10 photos to reduce the load on the servers

as long as we dont have 100s accepted at once I am fine with that ...

Robin



ABC
User currently offlineDC10Tim From United Kingdom, joined Jan 2005, 1406 posts, RR: 15
Reply 6, posted (8 years 2 months 2 weeks 18 hours ago) and read 4955 times:

Quoting Administrator (Thread starter):
It should all be sorted out now and the effected photographers have been notified to upload again with priority screening.

Hi Johan,

Further to your email received on Saturday evening, I re-uploaded one of the lost photos and left a note for the screeners and also emailed them, yet nothing has moved so far. I appreciate things don't happen like magic, but in the past when I've asked for a priority screen, the shot has been shown as being "in screening" within a couple of hours. It's not as though my life depends on it, but it is a little frustrating given it's nearly two weeks since it was first uploaded.

Regards,

Tim.



Obviously missing something....
User currently offlineFrippe From Sweden, joined Sep 2005, 183 posts, RR: 8
Reply 7, posted (8 years 2 months 2 weeks 16 hours ago) and read 4903 times:

Hello,
It is good news to hear that problems now seem to be solved. But I would like to know if we have to do anything special to have our lost photos screened with priority screening.

Maybe I ought to know how to fix a priority screening, but all my photos hitherto have never been near such an honor, so I simply don't know how to fix that.

Maybe the staff when things have calmed down a bit will pick up the lost and reuploaded ones from the queue and add them with priority?

Thanks in advance for any clarification,
Frippe


User currently offlineGary2880 From , joined Dec 1969, posts, RR:
Reply 8, posted (8 years 2 months 2 weeks 12 hours ago) and read 4779 times:

Quoting IL76 (Reply 4):
No, 10 photos (not just yours) need to have been accepted. If I screen a set of 10 photos, and as a result 5 photos have been rated HQ 3 times, these 5 will be added when at least 5 others have also been accepted.
So additions are done in batches of at least 10 photos to reduce the load on the servers. It's a technical thing, it has nothing to do with the screening process.

i see, cheers. thought it was a photographer to photographer thing.

at the rate im going if i had to wait till i had 10 accepted before they showed up in the database it would make a nice christmas present.


Top Of Page
Forum Index

This topic is archived and can not be replied to any more.

Printer friendly format

Similar topics:More similar topics...
How To Center The Plane? posted Fri May 12 2006 12:04:35 by Dcrusafon
How To Make The Most Of Paint Shop Pro 8 posted Fri Aug 5 2005 00:29:17 by ZSOFN
How To Link Photos Hosted At Airliners.net posted Thu Jul 21 2005 08:11:32 by Mfz
How To Remove Photos From Database posted Tue Jan 25 2005 17:22:03 by MauriceB
A Thread To Prove The Screeners Are Fair posted Mon Oct 25 2004 16:07:06 by Pepef
Good To See The Screeners Are Appreciated! posted Fri Aug 20 2004 10:38:26 by Skymonster
Have They Permission To Use The Following Photos.. posted Wed May 19 2004 18:36:45 by Karlok
How Do I Switch My Name On Photos In The DB? posted Tue Nov 4 2003 23:29:52 by Sushka
How To Level The Horizon posted Mon Mar 24 2003 15:49:20 by Jcs17
How To Improve The Quality? posted Wed Mar 5 2003 20:12:26 by MD