[Micronet] SPA bDrive for large archive of small data files

classic Classic list List threaded Threaded
16 messages Options
Reply | Threaded
Open this post in threaded view
|

[Micronet] SPA bDrive for large archive of small data files

Greg Merritt
Hello Micronet,

We'd love to use our SPA bDrive as an archive location for a infrequently and selectively accessed Bag of Holding for data sets of millions of small, individual, already-compressed ~10K files.

Folder drag & drop into the bDrive browser interface on Mac Chrome leads to unhappiness, as the browser crashes after a while.

Rather than tar-ing them, or dragging them over in subgroups, is there a more direct way to send over the lot? We'd rather not do the desktop synching with these, unless there's a super-elegant way to use that mechanism.

(Would *really* love to do an additive rsync here in the long run, but I'm probably just dreaming here.)

Thanks for any tips!

-Greg

 
-------------------------------------------------------------------------
The following was automatically added to this message by the list server:

To learn more about Micronet, including how to subscribe to or unsubscribe from its mailing list and how to find out about upcoming meetings, please visit the Micronet Web site:

http://micronet.berkeley.edu

Messages you send to this mailing list are public and world-viewable, and the list's archives can be browsed and searched on the Internet.  This means these messages can be viewed by (among others) your bosses, prospective employers, and people who have known you in the past.

ANNOUNCEMENTS: To send announcements to the Micronet list, please use the [hidden email] list.
Reply | Threaded
Open this post in threaded view
|

Re: [Micronet] SPA bDrive for large archive of small data files

Michael Chung
Hi Greg,

Have you checked this out?


Full disclaimer, I haven't used it myself, but it appears you can run incremental transfers, which should at least give you "rsync"-like functionality after an initial Full backup.

Thanks,

Michael Chung
Systems Administrator
Enterprise Computing & Service Management
Haas School of Business, UC Berkeley
Student Services Building, Room S300D
Berkeley, CA 94720-1900
 
Typical Office Schedule
Offsite: M-F
At Haas: On-demand

On Mon, Oct 5, 2015 at 4:23 PM, Greg Merritt <[hidden email]> wrote:
Hello Micronet,

We'd love to use our SPA bDrive as an archive location for a infrequently and selectively accessed Bag of Holding for data sets of millions of small, individual, already-compressed ~10K files.

Folder drag & drop into the bDrive browser interface on Mac Chrome leads to unhappiness, as the browser crashes after a while.

Rather than tar-ing them, or dragging them over in subgroups, is there a more direct way to send over the lot? We'd rather not do the desktop synching with these, unless there's a super-elegant way to use that mechanism.

(Would *really* love to do an additive rsync here in the long run, but I'm probably just dreaming here.)

Thanks for any tips!

-Greg


-------------------------------------------------------------------------
The following was automatically added to this message by the list server:

To learn more about Micronet, including how to subscribe to or unsubscribe from its mailing list and how to find out about upcoming meetings, please visit the Micronet Web site:

http://micronet.berkeley.edu

Messages you send to this mailing list are public and world-viewable, and the list's archives can be browsed and searched on the Internet.  This means these messages can be viewed by (among others) your bosses, prospective employers, and people who have known you in the past.

ANNOUNCEMENTS: To send announcements to the Micronet list, please use the [hidden email] list.



 
-------------------------------------------------------------------------
The following was automatically added to this message by the list server:

To learn more about Micronet, including how to subscribe to or unsubscribe from its mailing list and how to find out about upcoming meetings, please visit the Micronet Web site:

http://micronet.berkeley.edu

Messages you send to this mailing list are public and world-viewable, and the list's archives can be browsed and searched on the Internet.  This means these messages can be viewed by (among others) your bosses, prospective employers, and people who have known you in the past.

ANNOUNCEMENTS: To send announcements to the Micronet list, please use the [hidden email] list.
Reply | Threaded
Open this post in threaded view
|

Re: [Micronet] SPA bDrive for large archive of small data files

Greg Merritt
Ooh, no! I'll check it out. This may indeed be helpful after we get the historical backlog up there.

Thanks!

-Greg

On Mon, Oct 5, 2015 at 4:40 PM, Michael Chung <[hidden email]> wrote:
Hi Greg,

Have you checked this out?


Full disclaimer, I haven't used it myself, but it appears you can run incremental transfers, which should at least give you "rsync"-like functionality after an initial Full backup.

Thanks,

Michael Chung
Systems Administrator
Enterprise Computing & Service Management
Haas School of Business, UC Berkeley
Student Services Building, Room S300D
Berkeley, CA 94720-1900
 
Typical Office Schedule
Offsite: M-F
At Haas: On-demand

On Mon, Oct 5, 2015 at 4:23 PM, Greg Merritt <[hidden email]> wrote:
Hello Micronet,

We'd love to use our SPA bDrive as an archive location for a infrequently and selectively accessed Bag of Holding for data sets of millions of small, individual, already-compressed ~10K files.

Folder drag & drop into the bDrive browser interface on Mac Chrome leads to unhappiness, as the browser crashes after a while.

Rather than tar-ing them, or dragging them over in subgroups, is there a more direct way to send over the lot? We'd rather not do the desktop synching with these, unless there's a super-elegant way to use that mechanism.

(Would *really* love to do an additive rsync here in the long run, but I'm probably just dreaming here.)

Thanks for any tips!

-Greg


-------------------------------------------------------------------------
The following was automatically added to this message by the list server:

To learn more about Micronet, including how to subscribe to or unsubscribe from its mailing list and how to find out about upcoming meetings, please visit the Micronet Web site:

http://micronet.berkeley.edu

Messages you send to this mailing list are public and world-viewable, and the list's archives can be browsed and searched on the Internet.  This means these messages can be viewed by (among others) your bosses, prospective employers, and people who have known you in the past.

ANNOUNCEMENTS: To send announcements to the Micronet list, please use the [hidden email] list.




 
-------------------------------------------------------------------------
The following was automatically added to this message by the list server:

To learn more about Micronet, including how to subscribe to or unsubscribe from its mailing list and how to find out about upcoming meetings, please visit the Micronet Web site:

http://micronet.berkeley.edu

Messages you send to this mailing list are public and world-viewable, and the list's archives can be browsed and searched on the Internet.  This means these messages can be viewed by (among others) your bosses, prospective employers, and people who have known you in the past.

ANNOUNCEMENTS: To send announcements to the Micronet list, please use the [hidden email] list.
Reply | Threaded
Open this post in threaded view
|

Re: [Micronet] SPA bDrive for large archive of small data files

Chris Hoffman-2
Hi Greg, we have also seen people use the Box API to script file synchronization. I am not sure where that documentation is right now let me know if that is an option. 
Chris

On Oct 5, 2015, at 4:43 PM, Greg Merritt <[hidden email]> wrote:

Ooh, no! I'll check it out. This may indeed be helpful after we get the historical backlog up there.

Thanks!

-Greg

On Mon, Oct 5, 2015 at 4:40 PM, Michael Chung <[hidden email]> wrote:
Hi Greg,

Have you checked this out?


Full disclaimer, I haven't used it myself, but it appears you can run incremental transfers, which should at least give you "rsync"-like functionality after an initial Full backup.

Thanks,

Michael Chung
Systems Administrator
Enterprise Computing & Service Management
Haas School of Business, UC Berkeley
Student Services Building, Room S300D
Berkeley, CA 94720-1900
 
Typical Office Schedule
Offsite: M-F
At Haas: On-demand

On Mon, Oct 5, 2015 at 4:23 PM, Greg Merritt <[hidden email]> wrote:
Hello Micronet,

We'd love to use our SPA bDrive as an archive location for a infrequently and selectively accessed Bag of Holding for data sets of millions of small, individual, already-compressed ~10K files.

Folder drag & drop into the bDrive browser interface on Mac Chrome leads to unhappiness, as the browser crashes after a while.

Rather than tar-ing them, or dragging them over in subgroups, is there a more direct way to send over the lot? We'd rather not do the desktop synching with these, unless there's a super-elegant way to use that mechanism.

(Would *really* love to do an additive rsync here in the long run, but I'm probably just dreaming here.)

Thanks for any tips!

-Greg


-------------------------------------------------------------------------
The following was automatically added to this message by the list server:

To learn more about Micronet, including how to subscribe to or unsubscribe from its mailing list and how to find out about upcoming meetings, please visit the Micronet Web site:

http://micronet.berkeley.edu

Messages you send to this mailing list are public and world-viewable, and the list's archives can be browsed and searched on the Internet.  This means these messages can be viewed by (among others) your bosses, prospective employers, and people who have known you in the past.

ANNOUNCEMENTS: To send announcements to the Micronet list, please use the [hidden email] list.




-------------------------------------------------------------------------
The following was automatically added to this message by the list server:

To learn more about Micronet, including how to subscribe to or unsubscribe from its mailing list and how to find out about upcoming meetings, please visit the Micronet Web site:

http://micronet.berkeley.edu

Messages you send to this mailing list are public and world-viewable, and the list's archives can be browsed and searched on the Internet.  This means these messages can be viewed by (among others) your bosses, prospective employers, and people who have known you in the past.

ANNOUNCEMENTS: To send announcements to the Micronet list, please use the [hidden email] list.

 
-------------------------------------------------------------------------
The following was automatically added to this message by the list server:

To learn more about Micronet, including how to subscribe to or unsubscribe from its mailing list and how to find out about upcoming meetings, please visit the Micronet Web site:

http://micronet.berkeley.edu

Messages you send to this mailing list are public and world-viewable, and the list's archives can be browsed and searched on the Internet.  This means these messages can be viewed by (among others) your bosses, prospective employers, and people who have known you in the past.

ANNOUNCEMENTS: To send announcements to the Micronet list, please use the [hidden email] list.
Reply | Threaded
Open this post in threaded view
|

Re: [Micronet] SPA bDrive for large archive of small data files

Greg Merritt
Interesting.

Oh! I see that Box accounts now have unlimited capacity, too, yeah? (Guess I'd missed that.)

Thanks, I'll take a look! If you have documentation handy, do please toss my way. (This isn't super-duper time critical....well, until another hard drive dies......)

Thanks!

-Greg

On Mon, Oct 5, 2015 at 4:46 PM, Christopher R. HOFFMAN <[hidden email]> wrote:
Hi Greg, we have also seen people use the Box API to script file synchronization. I am not sure where that documentation is right now let me know if that is an option. 
Chris

On Oct 5, 2015, at 4:43 PM, Greg Merritt <[hidden email]> wrote:

Ooh, no! I'll check it out. This may indeed be helpful after we get the historical backlog up there.

Thanks!

-Greg

On Mon, Oct 5, 2015 at 4:40 PM, Michael Chung <[hidden email]> wrote:
Hi Greg,

Have you checked this out?


Full disclaimer, I haven't used it myself, but it appears you can run incremental transfers, which should at least give you "rsync"-like functionality after an initial Full backup.

Thanks,

Michael Chung
Systems Administrator
Enterprise Computing & Service Management
Haas School of Business, UC Berkeley
Student Services Building, Room S300D
Berkeley, CA 94720-1900
 
Typical Office Schedule
Offsite: M-F
At Haas: On-demand

On Mon, Oct 5, 2015 at 4:23 PM, Greg Merritt <[hidden email]> wrote:
Hello Micronet,

We'd love to use our SPA bDrive as an archive location for a infrequently and selectively accessed Bag of Holding for data sets of millions of small, individual, already-compressed ~10K files.

Folder drag & drop into the bDrive browser interface on Mac Chrome leads to unhappiness, as the browser crashes after a while.

Rather than tar-ing them, or dragging them over in subgroups, is there a more direct way to send over the lot? We'd rather not do the desktop synching with these, unless there's a super-elegant way to use that mechanism.

(Would *really* love to do an additive rsync here in the long run, but I'm probably just dreaming here.)

Thanks for any tips!

-Greg


-------------------------------------------------------------------------
The following was automatically added to this message by the list server:

To learn more about Micronet, including how to subscribe to or unsubscribe from its mailing list and how to find out about upcoming meetings, please visit the Micronet Web site:

http://micronet.berkeley.edu

Messages you send to this mailing list are public and world-viewable, and the list's archives can be browsed and searched on the Internet.  This means these messages can be viewed by (among others) your bosses, prospective employers, and people who have known you in the past.

ANNOUNCEMENTS: To send announcements to the Micronet list, please use the [hidden email] list.




-------------------------------------------------------------------------
The following was automatically added to this message by the list server:

To learn more about Micronet, including how to subscribe to or unsubscribe from its mailing list and how to find out about upcoming meetings, please visit the Micronet Web site:

http://micronet.berkeley.edu

Messages you send to this mailing list are public and world-viewable, and the list's archives can be browsed and searched on the Internet.  This means these messages can be viewed by (among others) your bosses, prospective employers, and people who have known you in the past.

ANNOUNCEMENTS: To send announcements to the Micronet list, please use the [hidden email] list.


 
-------------------------------------------------------------------------
The following was automatically added to this message by the list server:

To learn more about Micronet, including how to subscribe to or unsubscribe from its mailing list and how to find out about upcoming meetings, please visit the Micronet Web site:

http://micronet.berkeley.edu

Messages you send to this mailing list are public and world-viewable, and the list's archives can be browsed and searched on the Internet.  This means these messages can be viewed by (among others) your bosses, prospective employers, and people who have known you in the past.

ANNOUNCEMENTS: To send announcements to the Micronet list, please use the [hidden email] list.
Reply | Threaded
Open this post in threaded view
|

Re: [Micronet] SPA bDrive for large archive of small data files

Richard DeShong-2
In reply to this post by Greg Merritt
Hi Greg,
It seems like Google Drive is exactly what you want.  Install it, move all your files into the folder, and GD will churn away in the background uploading all of the files.  Moving forward, just drop new files in, and GD will sync those.

I have, in the past, seen forum issues with the sync index size - that is, when the file count gets too large, the sync process fails.  So you'd need to research that first.

On Mon, Oct 5, 2015 at 4:43 PM, Greg Merritt <[hidden email]> wrote:
Ooh, no! I'll check it out. This may indeed be helpful after we get the historical backlog up there.

Thanks!

-Greg

On Mon, Oct 5, 2015 at 4:40 PM, Michael Chung <[hidden email]> wrote:
Hi Greg,

Have you checked this out?


Full disclaimer, I haven't used it myself, but it appears you can run incremental transfers, which should at least give you "rsync"-like functionality after an initial Full backup.

Thanks,

Michael Chung
Systems Administrator
Enterprise Computing & Service Management
Haas School of Business, UC Berkeley
Student Services Building, Room S300D
Berkeley, CA 94720-1900
 
Typical Office Schedule
Offsite: M-F
At Haas: On-demand

On Mon, Oct 5, 2015 at 4:23 PM, Greg Merritt <[hidden email]> wrote:
Hello Micronet,

We'd love to use our SPA bDrive as an archive location for a infrequently and selectively accessed Bag of Holding for data sets of millions of small, individual, already-compressed ~10K files.

Folder drag & drop into the bDrive browser interface on Mac Chrome leads to unhappiness, as the browser crashes after a while.

Rather than tar-ing them, or dragging them over in subgroups, is there a more direct way to send over the lot? We'd rather not do the desktop synching with these, unless there's a super-elegant way to use that mechanism.

(Would *really* love to do an additive rsync here in the long run, but I'm probably just dreaming here.)

Thanks for any tips!

-Greg


-------------------------------------------------------------------------
The following was automatically added to this message by the list server:

To learn more about Micronet, including how to subscribe to or unsubscribe from its mailing list and how to find out about upcoming meetings, please visit the Micronet Web site:

http://micronet.berkeley.edu

Messages you send to this mailing list are public and world-viewable, and the list's archives can be browsed and searched on the Internet.  This means these messages can be viewed by (among others) your bosses, prospective employers, and people who have known you in the past.

ANNOUNCEMENTS: To send announcements to the Micronet list, please use the [hidden email] list.





-------------------------------------------------------------------------
The following was automatically added to this message by the list server:

To learn more about Micronet, including how to subscribe to or unsubscribe from its mailing list and how to find out about upcoming meetings, please visit the Micronet Web site:

http://micronet.berkeley.edu

Messages you send to this mailing list are public and world-viewable, and the list's archives can be browsed and searched on the Internet.  This means these messages can be viewed by (among others) your bosses, prospective employers, and people who have known you in the past.

ANNOUNCEMENTS: To send announcements to the Micronet list, please use the [hidden email] list.




--
Richard DeShong, Systems Analyst, Athletic Study Center, U.C.Berkeley
164 Chavez Student Center, Berkeley, CA, 94720-4220
510-642-5123     asc.berkeley.edu


 
-------------------------------------------------------------------------
The following was automatically added to this message by the list server:

To learn more about Micronet, including how to subscribe to or unsubscribe from its mailing list and how to find out about upcoming meetings, please visit the Micronet Web site:

http://micronet.berkeley.edu

Messages you send to this mailing list are public and world-viewable, and the list's archives can be browsed and searched on the Internet.  This means these messages can be viewed by (among others) your bosses, prospective employers, and people who have known you in the past.

ANNOUNCEMENTS: To send announcements to the Micronet list, please use the [hidden email] list.
Reply | Threaded
Open this post in threaded view
|

Re: [Micronet] SPA bDrive for large archive of small data files

Greg Merritt
Too much data to stick in a local gDrive folder, really...want to have no local long-term archive on a random desktop, if you dig....maybe for a one-time initial backfill, but then would want to purge from local....hmmm.....

Thanks for the tip on the sync index size!

-Greg

On Mon, Oct 5, 2015 at 4:54 PM, Richard DESHONG <[hidden email]> wrote:
Hi Greg,
It seems like Google Drive is exactly what you want.  Install it, move all your files into the folder, and GD will churn away in the background uploading all of the files.  Moving forward, just drop new files in, and GD will sync those.

I have, in the past, seen forum issues with the sync index size - that is, when the file count gets too large, the sync process fails.  So you'd need to research that first.

On Mon, Oct 5, 2015 at 4:43 PM, Greg Merritt <[hidden email]> wrote:
Ooh, no! I'll check it out. This may indeed be helpful after we get the historical backlog up there.

Thanks!

-Greg

On Mon, Oct 5, 2015 at 4:40 PM, Michael Chung <[hidden email]> wrote:
Hi Greg,

Have you checked this out?


Full disclaimer, I haven't used it myself, but it appears you can run incremental transfers, which should at least give you "rsync"-like functionality after an initial Full backup.

Thanks,

Michael Chung
Systems Administrator
Enterprise Computing & Service Management
Haas School of Business, UC Berkeley
Student Services Building, Room S300D
Berkeley, CA 94720-1900
 
Typical Office Schedule
Offsite: M-F
At Haas: On-demand

On Mon, Oct 5, 2015 at 4:23 PM, Greg Merritt <[hidden email]> wrote:
Hello Micronet,

We'd love to use our SPA bDrive as an archive location for a infrequently and selectively accessed Bag of Holding for data sets of millions of small, individual, already-compressed ~10K files.

Folder drag & drop into the bDrive browser interface on Mac Chrome leads to unhappiness, as the browser crashes after a while.

Rather than tar-ing them, or dragging them over in subgroups, is there a more direct way to send over the lot? We'd rather not do the desktop synching with these, unless there's a super-elegant way to use that mechanism.

(Would *really* love to do an additive rsync here in the long run, but I'm probably just dreaming here.)

Thanks for any tips!

-Greg


-------------------------------------------------------------------------
The following was automatically added to this message by the list server:

To learn more about Micronet, including how to subscribe to or unsubscribe from its mailing list and how to find out about upcoming meetings, please visit the Micronet Web site:

http://micronet.berkeley.edu

Messages you send to this mailing list are public and world-viewable, and the list's archives can be browsed and searched on the Internet.  This means these messages can be viewed by (among others) your bosses, prospective employers, and people who have known you in the past.

ANNOUNCEMENTS: To send announcements to the Micronet list, please use the [hidden email] list.





-------------------------------------------------------------------------
The following was automatically added to this message by the list server:

To learn more about Micronet, including how to subscribe to or unsubscribe from its mailing list and how to find out about upcoming meetings, please visit the Micronet Web site:

http://micronet.berkeley.edu

Messages you send to this mailing list are public and world-viewable, and the list's archives can be browsed and searched on the Internet.  This means these messages can be viewed by (among others) your bosses, prospective employers, and people who have known you in the past.

ANNOUNCEMENTS: To send announcements to the Micronet list, please use the [hidden email] list.




--
Richard DeShong, Systems Analyst, Athletic Study Center, U.C.Berkeley
164 Chavez Student Center, Berkeley, CA, 94720-4220
<a href="tel:510-642-5123" value="+15106425123" target="_blank">510-642-5123     asc.berkeley.edu



-------------------------------------------------------------------------
The following was automatically added to this message by the list server:

To learn more about Micronet, including how to subscribe to or unsubscribe from its mailing list and how to find out about upcoming meetings, please visit the Micronet Web site:

http://micronet.berkeley.edu

Messages you send to this mailing list are public and world-viewable, and the list's archives can be browsed and searched on the Internet.  This means these messages can be viewed by (among others) your bosses, prospective employers, and people who have known you in the past.

ANNOUNCEMENTS: To send announcements to the Micronet list, please use the [hidden email] list.



 
-------------------------------------------------------------------------
The following was automatically added to this message by the list server:

To learn more about Micronet, including how to subscribe to or unsubscribe from its mailing list and how to find out about upcoming meetings, please visit the Micronet Web site:

http://micronet.berkeley.edu

Messages you send to this mailing list are public and world-viewable, and the list's archives can be browsed and searched on the Internet.  This means these messages can be viewed by (among others) your bosses, prospective employers, and people who have known you in the past.

ANNOUNCEMENTS: To send announcements to the Micronet list, please use the [hidden email] list.
Reply | Threaded
Open this post in threaded view
|

Re: [Micronet] SPA bDrive for large archive of small data files

Richard DeShong-2
Sorry Greg, I guess I misunderstood your original post.  It sounded like you have a local library (was that the "Bag of Holding") that you wanted to put "into the cloud".  Once it is in the cloud (bDrive or Box, in this case), are you saying that you don't want to have the local library?

I was confused because you mentioned "additive rsync" which implies staff are putting the files into a local library and then you want to have those uploaded to the cloud.  At least, that's the way I saw it.

On Mon, Oct 5, 2015 at 5:02 PM, Greg Merritt <[hidden email]> wrote:
Too much data to stick in a local gDrive folder, really...want to have no local long-term archive on a random desktop, if you dig....maybe for a one-time initial backfill, but then would want to purge from local....hmmm.....

Thanks for the tip on the sync index size!

-Greg

On Mon, Oct 5, 2015 at 4:54 PM, Richard DESHONG <[hidden email]> wrote:
Hi Greg,
It seems like Google Drive is exactly what you want.  Install it, move all your files into the folder, and GD will churn away in the background uploading all of the files.  Moving forward, just drop new files in, and GD will sync those.

I have, in the past, seen forum issues with the sync index size - that is, when the file count gets too large, the sync process fails.  So you'd need to research that first.

On Mon, Oct 5, 2015 at 4:43 PM, Greg Merritt <[hidden email]> wrote:
Ooh, no! I'll check it out. This may indeed be helpful after we get the historical backlog up there.

Thanks!

-Greg

On Mon, Oct 5, 2015 at 4:40 PM, Michael Chung <[hidden email]> wrote:
Hi Greg,

Have you checked this out?


Full disclaimer, I haven't used it myself, but it appears you can run incremental transfers, which should at least give you "rsync"-like functionality after an initial Full backup.

Thanks,

Michael Chung
Systems Administrator
Enterprise Computing & Service Management
Haas School of Business, UC Berkeley
Student Services Building, Room S300D
Berkeley, CA 94720-1900
 
Typical Office Schedule
Offsite: M-F
At Haas: On-demand

On Mon, Oct 5, 2015 at 4:23 PM, Greg Merritt <[hidden email]> wrote:
Hello Micronet,

We'd love to use our SPA bDrive as an archive location for a infrequently and selectively accessed Bag of Holding for data sets of millions of small, individual, already-compressed ~10K files.

Folder drag & drop into the bDrive browser interface on Mac Chrome leads to unhappiness, as the browser crashes after a while.

Rather than tar-ing them, or dragging them over in subgroups, is there a more direct way to send over the lot? We'd rather not do the desktop synching with these, unless there's a super-elegant way to use that mechanism.

(Would *really* love to do an additive rsync here in the long run, but I'm probably just dreaming here.)

Thanks for any tips!

-Greg


-------------------------------------------------------------------------
The following was automatically added to this message by the list server:

To learn more about Micronet, including how to subscribe to or unsubscribe from its mailing list and how to find out about upcoming meetings, please visit the Micronet Web site:

http://micronet.berkeley.edu

Messages you send to this mailing list are public and world-viewable, and the list's archives can be browsed and searched on the Internet.  This means these messages can be viewed by (among others) your bosses, prospective employers, and people who have known you in the past.

ANNOUNCEMENTS: To send announcements to the Micronet list, please use the [hidden email] list.





-------------------------------------------------------------------------
The following was automatically added to this message by the list server:

To learn more about Micronet, including how to subscribe to or unsubscribe from its mailing list and how to find out about upcoming meetings, please visit the Micronet Web site:

http://micronet.berkeley.edu

Messages you send to this mailing list are public and world-viewable, and the list's archives can be browsed and searched on the Internet.  This means these messages can be viewed by (among others) your bosses, prospective employers, and people who have known you in the past.

ANNOUNCEMENTS: To send announcements to the Micronet list, please use the [hidden email] list.




--
Richard DeShong, Systems Analyst, Athletic Study Center, U.C.Berkeley
164 Chavez Student Center, Berkeley, CA, 94720-4220
<a href="tel:510-642-5123" value="+15106425123" target="_blank">510-642-5123     asc.berkeley.edu



-------------------------------------------------------------------------
The following was automatically added to this message by the list server:

To learn more about Micronet, including how to subscribe to or unsubscribe from its mailing list and how to find out about upcoming meetings, please visit the Micronet Web site:

http://micronet.berkeley.edu

Messages you send to this mailing list are public and world-viewable, and the list's archives can be browsed and searched on the Internet.  This means these messages can be viewed by (among others) your bosses, prospective employers, and people who have known you in the past.

ANNOUNCEMENTS: To send announcements to the Micronet list, please use the [hidden email] list.





--
Richard DeShong, Systems Analyst, Athletic Study Center, U.C.Berkeley
164 Chavez Student Center, Berkeley, CA, 94720-4220
510-642-5123     asc.berkeley.edu


 
-------------------------------------------------------------------------
The following was automatically added to this message by the list server:

To learn more about Micronet, including how to subscribe to or unsubscribe from its mailing list and how to find out about upcoming meetings, please visit the Micronet Web site:

http://micronet.berkeley.edu

Messages you send to this mailing list are public and world-viewable, and the list's archives can be browsed and searched on the Internet.  This means these messages can be viewed by (among others) your bosses, prospective employers, and people who have known you in the past.

ANNOUNCEMENTS: To send announcements to the Micronet list, please use the [hidden email] list.
Reply | Threaded
Open this post in threaded view
|

Re: [Micronet] SPA bDrive for large archive of small data files

Richard DeShong-2
In reply to this post by Greg Merritt
This is from community.box.com, dated 12-16-2014:

Unsupported Use Cases: 
We do not support syncing more than 40,000 files. The sync client works optimally when around 10,000 files are being synced. 

And on the Google Drive forum, there is a long discussion that has run for 2 yrs (ending last month) of people having issues syncing large numbers of files (seems like most of them are using the Windows version of Drive, so OSX results might be different).

On Mon, Oct 5, 2015 at 4:23 PM, Greg Merritt <[hidden email]> wrote:
Hello Micronet,

We'd love to use our SPA bDrive as an archive location for a infrequently and selectively accessed Bag of Holding for data sets of millions of small, individual, already-compressed ~10K files.

Folder drag & drop into the bDrive browser interface on Mac Chrome leads to unhappiness, as the browser crashes after a while.

Rather than tar-ing them, or dragging them over in subgroups, is there a more direct way to send over the lot? We'd rather not do the desktop synching with these, unless there's a super-elegant way to use that mechanism.

(Would *really* love to do an additive rsync here in the long run, but I'm probably just dreaming here.)

Thanks for any tips!

-Greg


-------------------------------------------------------------------------
The following was automatically added to this message by the list server:

To learn more about Micronet, including how to subscribe to or unsubscribe from its mailing list and how to find out about upcoming meetings, please visit the Micronet Web site:

http://micronet.berkeley.edu

Messages you send to this mailing list are public and world-viewable, and the list's archives can be browsed and searched on the Internet.  This means these messages can be viewed by (among others) your bosses, prospective employers, and people who have known you in the past.

ANNOUNCEMENTS: To send announcements to the Micronet list, please use the [hidden email] list.




--
Richard DeShong, Systems Analyst, Athletic Study Center, U.C.Berkeley
164 Chavez Student Center, Berkeley, CA, 94720-4220
510-642-5123     asc.berkeley.edu


 
-------------------------------------------------------------------------
The following was automatically added to this message by the list server:

To learn more about Micronet, including how to subscribe to or unsubscribe from its mailing list and how to find out about upcoming meetings, please visit the Micronet Web site:

http://micronet.berkeley.edu

Messages you send to this mailing list are public and world-viewable, and the list's archives can be browsed and searched on the Internet.  This means these messages can be viewed by (among others) your bosses, prospective employers, and people who have known you in the past.

ANNOUNCEMENTS: To send announcements to the Micronet list, please use the [hidden email] list.
Reply | Threaded
Open this post in threaded view
|

[Micronet] SPA bDrive for large archive of small data files

Greg Merritt
In reply to this post by Richard DeShong-2
Definitely no need to apologize -- I really haven't explained very well.

We have some real-world data feeds that work on slightly different timescales -- some are daily reports, archived for a relatively long time remotely (but copied to campus daily), some are updated & polled every minute, and cache for a few days remotely, while another can be polled every minute or so, but only "current" data is ever available.

On our side, it is very useful to have deep histories available, so that folks can, say, get at a certain day or month of data from a day of interest in the past. Another application requires very-current data only, and doesn't care about anything more than a coup,e of minutes old.

"Bag of Holding" was a d&d reference -- a magical bag whose inside is much bigger than its outside. (Like a TARDIS.)

I should have written "rsync -av" in which files are never deleted from the destination. Anything new in the source gets copied to the destination, but he destination copy stays even if the source file disappears.

This data usually comes in on Linux systems, but I did copy a big chunk over to a Mac to get to bDrive.

-Greg

Le lundi 5 octobre 2015, Richard DESHONG <<a href="javascript:_e(%7B%7D,&#39;cvml&#39;,&#39;rdeshong@berkeley.edu&#39;);" target="_blank">rdeshong@...> a écrit :
Sorry Greg, I guess I misunderstood your original post.  It sounded like you have a local library (was that the "Bag of Holding") that you wanted to put "into the cloud".  Once it is in the cloud (bDrive or Box, in this case), are you saying that you don't want to have the local library?

I was confused because you mentioned "additive rsync" which implies staff are putting the files into a local library and then you want to have those uploaded to the cloud.  At least, that's the way I saw it.

On Mon, Oct 5, 2015 at 5:02 PM, Greg Merritt <[hidden email]> wrote:
Too much data to stick in a local gDrive folder, really...want to have no local long-term archive on a random desktop, if you dig....maybe for a one-time initial backfill, but then would want to purge from local....hmmm.....

Thanks for the tip on the sync index size!

-Greg

On Mon, Oct 5, 2015 at 4:54 PM, Richard DESHONG <[hidden email]> wrote:
Hi Greg,
It seems like Google Drive is exactly what you want.  Install it, move all your files into the folder, and GD will churn away in the background uploading all of the files.  Moving forward, just drop new files in, and GD will sync those.

I have, in the past, seen forum issues with the sync index size - that is, when the file count gets too large, the sync process fails.  So you'd need to research that first.

On Mon, Oct 5, 2015 at 4:43 PM, Greg Merritt <[hidden email]> wrote:
Ooh, no! I'll check it out. This may indeed be helpful after we get the historical backlog up there.

Thanks!

-Greg

On Mon, Oct 5, 2015 at 4:40 PM, Michael Chung <[hidden email]> wrote:
Hi Greg,

Have you checked this out?


Full disclaimer, I haven't used it myself, but it appears you can run incremental transfers, which should at least give you "rsync"-like functionality after an initial Full backup.

Thanks,

Michael Chung
Systems Administrator
Enterprise Computing & Service Management
Haas School of Business, UC Berkeley
Student Services Building, Room S300D
Berkeley, CA 94720-1900
 
Typical Office Schedule
Offsite: M-F
At Haas: On-demand

On Mon, Oct 5, 2015 at 4:23 PM, Greg Merritt <[hidden email]> wrote:
Hello Micronet,

We'd love to use our SPA bDrive as an archive location for a infrequently and selectively accessed Bag of Holding for data sets of millions of small, individual, already-compressed ~10K files.

Folder drag & drop into the bDrive browser interface on Mac Chrome leads to unhappiness, as the browser crashes after a while.

Rather than tar-ing them, or dragging them over in subgroups, is there a more direct way to send over the lot? We'd rather not do the desktop synching with these, unless there's a super-elegant way to use that mechanism.

(Would *really* love to do an additive rsync here in the long run, but I'm probably just dreaming here.)

Thanks for any tips!

-Greg


-------------------------------------------------------------------------
The following was automatically added to this message by the list server:

To learn more about Micronet, including how to subscribe to or unsubscribe from its mailing list and how to find out about upcoming meetings, please visit the Micronet Web site:

http://micronet.berkeley.edu

Messages you send to this mailing list are public and world-viewable, and the list's archives can be browsed and searched on the Internet.  This means these messages can be viewed by (among others) your bosses, prospective employers, and people who have known you in the past.

ANNOUNCEMENTS: To send announcements to the Micronet list, please use the [hidden email] list.





-------------------------------------------------------------------------
The following was automatically added to this message by the list server:

To learn more about Micronet, including how to subscribe to or unsubscribe from its mailing list and how to find out about upcoming meetings, please visit the Micronet Web site:

http://micronet.berkeley.edu

Messages you send to this mailing list are public and world-viewable, and the list's archives can be browsed and searched on the Internet.  This means these messages can be viewed by (among others) your bosses, prospective employers, and people who have known you in the past.

ANNOUNCEMENTS: To send announcements to the Micronet list, please use the [hidden email] list.




--
Richard DeShong, Systems Analyst, Athletic Study Center, U.C.Berkeley
164 Chavez Student Center, Berkeley, CA, 94720-4220
<a href="tel:510-642-5123" value="+15106425123" target="_blank">510-642-5123     asc.berkeley.edu



-------------------------------------------------------------------------
The following was automatically added to this message by the list server:

To learn more about Micronet, including how to subscribe to or unsubscribe from its mailing list and how to find out about upcoming meetings, please visit the Micronet Web site:

http://micronet.berkeley.edu

Messages you send to this mailing list are public and world-viewable, and the list's archives can be browsed and searched on the Internet.  This means these messages can be viewed by (among others) your bosses, prospective employers, and people who have known you in the past.

ANNOUNCEMENTS: To send announcements to the Micronet list, please use the [hidden email] list.





--
Richard DeShong, Systems Analyst, Athletic Study Center, U.C.Berkeley
164 Chavez Student Center, Berkeley, CA, 94720-4220
510-642-5123     asc.berkeley.edu



--
Envoyé avec Gmail Mobile

 
-------------------------------------------------------------------------
The following was automatically added to this message by the list server:

To learn more about Micronet, including how to subscribe to or unsubscribe from its mailing list and how to find out about upcoming meetings, please visit the Micronet Web site:

http://micronet.berkeley.edu

Messages you send to this mailing list are public and world-viewable, and the list's archives can be browsed and searched on the Internet.  This means these messages can be viewed by (among others) your bosses, prospective employers, and people who have known you in the past.

ANNOUNCEMENTS: To send announcements to the Micronet list, please use the [hidden email] list.
Reply | Threaded
Open this post in threaded view
|

Re: [Micronet] SPA bDrive for large archive of small data files

Greg Merritt
In reply to this post by Richard DeShong-2
Whoops! Ok....we either need to tar or rethink.  Thanks!!!

-Greg


Le lundi 5 octobre 2015, Richard DESHONG <[hidden email]> a écrit :
This is from community.box.com, dated 12-16-2014:

Unsupported Use Cases: 
We do not support syncing more than 40,000 files. The sync client works optimally when around 10,000 files are being synced. 

And on the Google Drive forum, there is a long discussion that has run for 2 yrs (ending last month) of people having issues syncing large numbers of files (seems like most of them are using the Windows version of Drive, so OSX results might be different).

On Mon, Oct 5, 2015 at 4:23 PM, Greg Merritt <<a href="javascript:_e(%7B%7D,&#39;cvml&#39;,&#39;gmerritt@berkeley.edu&#39;);" target="_blank">gmerritt@...> wrote:
Hello Micronet,

We'd love to use our SPA bDrive as an archive location for a infrequently and selectively accessed Bag of Holding for data sets of millions of small, individual, already-compressed ~10K files.

Folder drag & drop into the bDrive browser interface on Mac Chrome leads to unhappiness, as the browser crashes after a while.

Rather than tar-ing them, or dragging them over in subgroups, is there a more direct way to send over the lot? We'd rather not do the desktop synching with these, unless there's a super-elegant way to use that mechanism.

(Would *really* love to do an additive rsync here in the long run, but I'm probably just dreaming here.)

Thanks for any tips!

-Greg


-------------------------------------------------------------------------
The following was automatically added to this message by the list server:

To learn more about Micronet, including how to subscribe to or unsubscribe from its mailing list and how to find out about upcoming meetings, please visit the Micronet Web site:

http://micronet.berkeley.edu

Messages you send to this mailing list are public and world-viewable, and the list's archives can be browsed and searched on the Internet.  This means these messages can be viewed by (among others) your bosses, prospective employers, and people who have known you in the past.

ANNOUNCEMENTS: To send announcements to the Micronet list, please use the <a href="javascript:_e(%7B%7D,&#39;cvml&#39;,&#39;micronet-announce@lists.berkeley.edu&#39;);" target="_blank">micronet-announce@... list.




--
Richard DeShong, Systems Analyst, Athletic Study Center, U.C.Berkeley
164 Chavez Student Center, Berkeley, CA, 94720-4220
510-642-5123     asc.berkeley.edu



--
Envoyé avec Gmail Mobile

 
-------------------------------------------------------------------------
The following was automatically added to this message by the list server:

To learn more about Micronet, including how to subscribe to or unsubscribe from its mailing list and how to find out about upcoming meetings, please visit the Micronet Web site:

http://micronet.berkeley.edu

Messages you send to this mailing list are public and world-viewable, and the list's archives can be browsed and searched on the Internet.  This means these messages can be viewed by (among others) your bosses, prospective employers, and people who have known you in the past.

ANNOUNCEMENTS: To send announcements to the Micronet list, please use the [hidden email] list.
Reply | Threaded
Open this post in threaded view
|

Re: [Micronet] SPA bDrive for large archive of small data files

Daniel Companeetz

Hi Greg,

 

Is this a one time thing, or will you be creating this files often?

 

The IST-PCSSC offers a managed file transfer service that may be of use for the recurring runs of these. I do not think that it will be good to transfer millions of 10K files, as it may take too many resources from production transfers, but if it is of interest of you, ping me and let’s discuss it. Combined with the workload automation tool (a.k.a. job scheduling), you could have a set of jobs tar-ing, compressing, and transferring the files on a periodic basis (your additive rsync on enhancers J) without the need to do a lot of scripting, and with the addition that someone will be looking at the failures of the process.

 

For reference… http://community.box.com/t5/Managing-Your-Content/Using-Box-with-FTP/ta-p/312

 

For your initial upload, in the link they show a filezilla screenshot, which you could set on a computer and let run as long as it needs to. With filezilla, you can configure the bandwidth that you want to allocate so you will not be “killing” the network, and you can set concurrent uploads.

 

And just in case, bDrive is Google Drive based, while Box is Box…  For Box, you may need to contact the admin to create a non-person account.

 

Go to the web page for our team below, for more information.

 

Hope this helps!

 

Daniel Companeetz

Production Control Systems Administrator - Lead Architect

Infrastructure Services, IS&T

UC Berkeley

2195 Hearst Avenue #330B

Berkeley, CA 94720  

Office: +1 (510) 664-7480

Cell: +1 (510) 725-1790

email: [hidden email]

http://ist.berkeley.edu/is/platforms/pcssc

Scheduled Unavailability: 3/19/16 – 3/26/16

 

From: [hidden email] [mailto:[hidden email]] On Behalf Of Greg Merritt
Sent: Monday, October 05, 2015 7:55 PM
To: Richard DESHONG <[hidden email]>
Cc: [hidden email]
Subject: Re: [Micronet] SPA bDrive for large archive of small data files

 

Whoops! Ok....we either need to tar or rethink.  Thanks!!!

 

-Greg



Le lundi 5 octobre 2015, Richard DESHONG <[hidden email]> a écrit :

This is from community.box.com, dated 12-16-2014:

 

Unsupported Use Cases: 

We do not support syncing more than 40,000 files. The sync client works optimally when around 10,000 files are being synced. 

 

And on the Google Drive forum, there is a long discussion that has run for 2 yrs (ending last month) of people having issues syncing large numbers of files (seems like most of them are using the Windows version of Drive, so OSX results might be different).

 

On Mon, Oct 5, 2015 at 4:23 PM, Greg Merritt <<a href="javascript:_e(%7B%7D,&#39;cvml&#39;,&#39;gmerritt@berkeley.edu&#39;);" target="_blank">gmerritt@...> wrote:

Hello Micronet,

 

We'd love to use our SPA bDrive as an archive location for a infrequently and selectively accessed Bag of Holding for data sets of millions of small, individual, already-compressed ~10K files.

 

Folder drag & drop into the bDrive browser interface on Mac Chrome leads to unhappiness, as the browser crashes after a while.

 

Rather than tar-ing them, or dragging them over in subgroups, is there a more direct way to send over the lot? We'd rather not do the desktop synching with these, unless there's a super-elegant way to use that mechanism.

 

(Would *really* love to do an additive rsync here in the long run, but I'm probably just dreaming here.)

 

Thanks for any tips!

 

-Greg



-------------------------------------------------------------------------
The following was automatically added to this message by the list server:

To learn more about Micronet, including how to subscribe to or unsubscribe from its mailing list and how to find out about upcoming meetings, please visit the Micronet Web site:

http://micronet.berkeley.edu

Messages you send to this mailing list are public and world-viewable, and the list's archives can be browsed and searched on the Internet.  This means these messages can be viewed by (among others) your bosses, prospective employers, and people who have known you in the past.

ANNOUNCEMENTS: To send announcements to the Micronet list, please use the <a href="javascript:_e(%7B%7D,&#39;cvml&#39;,&#39;micronet-announce@lists.berkeley.edu&#39;);" target="_blank">micronet-announce@... list.



 

--

Richard DeShong, Systems Analyst, Athletic Study Center, U.C.Berkeley
164 Chavez Student Center, Berkeley, CA, 94720-4220
510-642-5123     asc.berkeley.edu



--
Envoyé avec Gmail Mobile


 
-------------------------------------------------------------------------
The following was automatically added to this message by the list server:

To learn more about Micronet, including how to subscribe to or unsubscribe from its mailing list and how to find out about upcoming meetings, please visit the Micronet Web site:

http://micronet.berkeley.edu

Messages you send to this mailing list are public and world-viewable, and the list's archives can be browsed and searched on the Internet.  This means these messages can be viewed by (among others) your bosses, prospective employers, and people who have known you in the past.

ANNOUNCEMENTS: To send announcements to the Micronet list, please use the [hidden email] list.
Reply | Threaded
Open this post in threaded view
|

Re: [Micronet] SPA bDrive for large archive of small data files

Richard DeShong-2
In reply to this post by Greg Merritt
And... the Box and Drive sync's are meant to be two-way, so that you could have one or more local mirrors of your cloud storage.

It sounds like what you're looking for is an archiving function.  Consider looking at backup software that can communicate with cloud storage.  There is probably one that works with Box or Google.

On Mon, Oct 5, 2015 at 7:55 PM, Greg Merritt <[hidden email]> wrote:
Whoops! Ok....we either need to tar or rethink.  Thanks!!!

-Greg


Le lundi 5 octobre 2015, Richard DESHONG <[hidden email]> a écrit :
This is from community.box.com, dated 12-16-2014:

Unsupported Use Cases: 
We do not support syncing more than 40,000 files. The sync client works optimally when around 10,000 files are being synced. 

And on the Google Drive forum, there is a long discussion that has run for 2 yrs (ending last month) of people having issues syncing large numbers of files (seems like most of them are using the Windows version of Drive, so OSX results might be different).

On Mon, Oct 5, 2015 at 4:23 PM, Greg Merritt <[hidden email]> wrote:
Hello Micronet,

We'd love to use our SPA bDrive as an archive location for a infrequently and selectively accessed Bag of Holding for data sets of millions of small, individual, already-compressed ~10K files.

Folder drag & drop into the bDrive browser interface on Mac Chrome leads to unhappiness, as the browser crashes after a while.

Rather than tar-ing them, or dragging them over in subgroups, is there a more direct way to send over the lot? We'd rather not do the desktop synching with these, unless there's a super-elegant way to use that mechanism.

(Would *really* love to do an additive rsync here in the long run, but I'm probably just dreaming here.)

Thanks for any tips!

-Greg


-------------------------------------------------------------------------
The following was automatically added to this message by the list server:

To learn more about Micronet, including how to subscribe to or unsubscribe from its mailing list and how to find out about upcoming meetings, please visit the Micronet Web site:

http://micronet.berkeley.edu

Messages you send to this mailing list are public and world-viewable, and the list's archives can be browsed and searched on the Internet.  This means these messages can be viewed by (among others) your bosses, prospective employers, and people who have known you in the past.

ANNOUNCEMENTS: To send announcements to the Micronet list, please use the [hidden email] list.




--
Richard DeShong, Systems Analyst, Athletic Study Center, U.C.Berkeley
164 Chavez Student Center, Berkeley, CA, 94720-4220
510-642-5123     asc.berkeley.edu



--
Envoyé avec Gmail Mobile



--
Richard DeShong, Systems Analyst, Athletic Study Center, U.C.Berkeley
164 Chavez Student Center, Berkeley, CA, 94720-4220
510-642-5123     asc.berkeley.edu


 
-------------------------------------------------------------------------
The following was automatically added to this message by the list server:

To learn more about Micronet, including how to subscribe to or unsubscribe from its mailing list and how to find out about upcoming meetings, please visit the Micronet Web site:

http://micronet.berkeley.edu

Messages you send to this mailing list are public and world-viewable, and the list's archives can be browsed and searched on the Internet.  This means these messages can be viewed by (among others) your bosses, prospective employers, and people who have known you in the past.

ANNOUNCEMENTS: To send announcements to the Micronet list, please use the [hidden email] list.
Reply | Threaded
Open this post in threaded view
|

Re: [Micronet] SPA bDrive for large archive of small data files

Greg Merritt
Yeah -- as Daniel references, there are two problems:

1. schlep the history into cloud
2. regularly, continuously add to it

We're massively getting out of the hardware business, so any secondary storage would also be cloud. No more "local" for us.

We also have a nice TB of IST NFS, which is totally cool, but too hard for regular users to access. Yes -- really. ;)

Thanks for all of the great ideas, everyone! I've got lots to chew on...before my [UU_U_U] raid6 dies... o.0

-Greg



On Tue, Oct 6, 2015 at 9:34 AM, Richard DESHONG <[hidden email]> wrote:
And... the Box and Drive sync's are meant to be two-way, so that you could have one or more local mirrors of your cloud storage.

It sounds like what you're looking for is an archiving function.  Consider looking at backup software that can communicate with cloud storage.  There is probably one that works with Box or Google.

On Mon, Oct 5, 2015 at 7:55 PM, Greg Merritt <[hidden email]> wrote:
Whoops! Ok....we either need to tar or rethink.  Thanks!!!

-Greg


Le lundi 5 octobre 2015, Richard DESHONG <[hidden email]> a écrit :
This is from community.box.com, dated 12-16-2014:

Unsupported Use Cases: 
We do not support syncing more than 40,000 files. The sync client works optimally when around 10,000 files are being synced. 

And on the Google Drive forum, there is a long discussion that has run for 2 yrs (ending last month) of people having issues syncing large numbers of files (seems like most of them are using the Windows version of Drive, so OSX results might be different).

On Mon, Oct 5, 2015 at 4:23 PM, Greg Merritt <[hidden email]> wrote:
Hello Micronet,

We'd love to use our SPA bDrive as an archive location for a infrequently and selectively accessed Bag of Holding for data sets of millions of small, individual, already-compressed ~10K files.

Folder drag & drop into the bDrive browser interface on Mac Chrome leads to unhappiness, as the browser crashes after a while.

Rather than tar-ing them, or dragging them over in subgroups, is there a more direct way to send over the lot? We'd rather not do the desktop synching with these, unless there's a super-elegant way to use that mechanism.

(Would *really* love to do an additive rsync here in the long run, but I'm probably just dreaming here.)

Thanks for any tips!

-Greg


-------------------------------------------------------------------------
The following was automatically added to this message by the list server:

To learn more about Micronet, including how to subscribe to or unsubscribe from its mailing list and how to find out about upcoming meetings, please visit the Micronet Web site:

http://micronet.berkeley.edu

Messages you send to this mailing list are public and world-viewable, and the list's archives can be browsed and searched on the Internet.  This means these messages can be viewed by (among others) your bosses, prospective employers, and people who have known you in the past.

ANNOUNCEMENTS: To send announcements to the Micronet list, please use the [hidden email] list.




--
Richard DeShong, Systems Analyst, Athletic Study Center, U.C.Berkeley
164 Chavez Student Center, Berkeley, CA, 94720-4220
<a href="tel:510-642-5123" value="+15106425123" target="_blank">510-642-5123     asc.berkeley.edu



--
Envoyé avec Gmail Mobile



--
Richard DeShong, Systems Analyst, Athletic Study Center, U.C.Berkeley
164 Chavez Student Center, Berkeley, CA, 94720-4220
<a href="tel:510-642-5123" value="+15106425123" target="_blank">510-642-5123     asc.berkeley.edu



 
-------------------------------------------------------------------------
The following was automatically added to this message by the list server:

To learn more about Micronet, including how to subscribe to or unsubscribe from its mailing list and how to find out about upcoming meetings, please visit the Micronet Web site:

http://micronet.berkeley.edu

Messages you send to this mailing list are public and world-viewable, and the list's archives can be browsed and searched on the Internet.  This means these messages can be viewed by (among others) your bosses, prospective employers, and people who have known you in the past.

ANNOUNCEMENTS: To send announcements to the Micronet list, please use the [hidden email] list.
Reply | Threaded
Open this post in threaded view
|

Re: [Micronet] SPA bDrive for large archive of small data files

Greg Merritt
As an aside -- while unlikely and uncommon, my group's research nature means that there can sometimes be hiccups in funding...so, for us, subsidized solutions like bDrive and Box have a strong appeal to us along the business continuity axis!

-Greg

 
-------------------------------------------------------------------------
The following was automatically added to this message by the list server:

To learn more about Micronet, including how to subscribe to or unsubscribe from its mailing list and how to find out about upcoming meetings, please visit the Micronet Web site:

http://micronet.berkeley.edu

Messages you send to this mailing list are public and world-viewable, and the list's archives can be browsed and searched on the Internet.  This means these messages can be viewed by (among others) your bosses, prospective employers, and people who have known you in the past.

ANNOUNCEMENTS: To send announcements to the Micronet list, please use the [hidden email] list.
Reply | Threaded
Open this post in threaded view
|

Re: [Micronet] SPA bDrive for large archive of small data files

Richard DeShong-2
In reply to this post by Richard DeShong-2
An added to note about this issue:  Box has as FTP server that you can use.  They recommend using it in batches of 150K files (or 250GB).  Sounds doable for the initial load.

Details:


On Tue, Oct 6, 2015 at 9:34 AM, Richard DESHONG <[hidden email]> wrote:
And... the Box and Drive sync's are meant to be two-way, so that you could have one or more local mirrors of your cloud storage.

It sounds like what you're looking for is an archiving function.  Consider looking at backup software that can communicate with cloud storage.  There is probably one that works with Box or Google.

On Mon, Oct 5, 2015 at 7:55 PM, Greg Merritt <[hidden email]> wrote:
Whoops! Ok....we either need to tar or rethink.  Thanks!!!

-Greg


Le lundi 5 octobre 2015, Richard DESHONG <[hidden email]> a écrit :
This is from community.box.com, dated 12-16-2014:

Unsupported Use Cases: 
We do not support syncing more than 40,000 files. The sync client works optimally when around 10,000 files are being synced. 

And on the Google Drive forum, there is a long discussion that has run for 2 yrs (ending last month) of people having issues syncing large numbers of files (seems like most of them are using the Windows version of Drive, so OSX results might be different).

On Mon, Oct 5, 2015 at 4:23 PM, Greg Merritt <[hidden email]> wrote:
Hello Micronet,

We'd love to use our SPA bDrive as an archive location for a infrequently and selectively accessed Bag of Holding for data sets of millions of small, individual, already-compressed ~10K files.

Folder drag & drop into the bDrive browser interface on Mac Chrome leads to unhappiness, as the browser crashes after a while.

Rather than tar-ing them, or dragging them over in subgroups, is there a more direct way to send over the lot? We'd rather not do the desktop synching with these, unless there's a super-elegant way to use that mechanism.

(Would *really* love to do an additive rsync here in the long run, but I'm probably just dreaming here.)

Thanks for any tips!

-Greg


-------------------------------------------------------------------------
The following was automatically added to this message by the list server:

To learn more about Micronet, including how to subscribe to or unsubscribe from its mailing list and how to find out about upcoming meetings, please visit the Micronet Web site:

http://micronet.berkeley.edu

Messages you send to this mailing list are public and world-viewable, and the list's archives can be browsed and searched on the Internet.  This means these messages can be viewed by (among others) your bosses, prospective employers, and people who have known you in the past.

ANNOUNCEMENTS: To send announcements to the Micronet list, please use the [hidden email] list.




--
Richard DeShong, Systems Analyst, Athletic Study Center, U.C.Berkeley
164 Chavez Student Center, Berkeley, CA, 94720-4220
510-642-5123     asc.berkeley.edu



--
Envoyé avec Gmail Mobile



--
Richard DeShong, Systems Analyst, Athletic Study Center, U.C.Berkeley
164 Chavez Student Center, Berkeley, CA, 94720-4220
510-642-5123     asc.berkeley.edu




--
Richard DeShong, Systems Analyst, Athletic Study Center, U.C.Berkeley
164 Chavez Student Center, Berkeley, CA, 94720-4220
510-642-5123     asc.berkeley.edu


 
-------------------------------------------------------------------------
The following was automatically added to this message by the list server:

To learn more about Micronet, including how to subscribe to or unsubscribe from its mailing list and how to find out about upcoming meetings, please visit the Micronet Web site:

http://micronet.berkeley.edu

Messages you send to this mailing list are public and world-viewable, and the list's archives can be browsed and searched on the Internet.  This means these messages can be viewed by (among others) your bosses, prospective employers, and people who have known you in the past.

ANNOUNCEMENTS: To send announcements to the Micronet list, please use the [hidden email] list.