Differences

This shows you the differences between two versions of the page.

--- file_storage [2022/09/04 23:41] – created wtkadmin
+++ file_storage [2023/08/23 20:51] (current) – [SQL Data] wtkadmin
@@ Line 11: / Line 11: @@
 ===== File Uploads =====
-The Wizard's Toolkit library has a PHP function that works in conjunction with our JS function to make uploading a file easy.  The **/demo** folder has several example files showing exactly how this is done.  The PHP code requires pulling the data from the SQL table like this;
+The Wizard's Toolkit library has a PHP function that works in conjunction with our JS function to make uploading a file easy.  The **/demo** folder has several example files showing exactly how this is done.  The PHP code requires pulling the data from the SQL table.
+===== SQL Data =====
+Storing images and documents as blobs in a SQL database is a horrible idea!  This makes backups take considerably longer, stresses the SQL DB by pulling the images out every time needed, plus several other reasons.  For example a DB backup I managed took 12 hours nightly due to images and documents stored in the DB as blobs.  Once it was changed to the Wizard's Toolkit method of storing the location of the file on the server, the nightly backups took less than an hour.
+In the data table the `FilePath` is stored in a separate data column than the `NewFileName`.  The reason it is called **New**FileName is because WTK renames the file during the upload process.  That way the length of the 'FileName' column only needs to be varchar(12).  The name is generated by inserting a row into the wtkGUID data table and taking the auto-incremented GUID value.  Then a 'w' is prepended and the original file extension is added.  If the file extension is **jpeg** it will be changed to **jpg**.
+For example an uploaded file may end up with a name like `w1087.png`.  Here is an example SQL statement to pull the FirstName and file-associated columns from the wtkUsers table (the FirstName is not relevant for file upload but likely to be needed on the page.
 <code sql>
@@ Line 40: / Line 48: @@
   * $fncShowOneClickUpload defaults to 'N' but if set to 'Y' then adds button to upload using AJAX without needing a 'Save' button
   * $fncAccept defaults to 'accept="image/*"'; you can change this to other document filters like accept=".pdf"
-  * $fncThumbnail defaults to 'N'; if set to 'Y' then adds an <img id="imgPreview" ...> which will show a preview of images
+  * $fncThumbnail defaults to 'N'; if set to 'Y' then adds an <img id="imgPreview" ...> which will show a preview of images but only if it is stored locally
+===== Public Files =====
+Files which are not sensitive and should be viewable by anyone on the website should be stored in a web folder that is available to the website.  For example a sub-folder named /imgs/ and then you can subcategorize by internal subfolders.  Like /imgs/users/ and /imgs/pets/ .  The above example PHP code shows an example of public file location.
+===== wtkFiles Data Table =====
+Wizard's Toolkit makes uploading files and tracking them easy with a combination of SQL data, JS and PHP functions.  Plus there is PHP integration with both AWS S3 and Cloudflare R2 (which use the same AWS SDK).  PHP pages are provided for managing uploading of files, storing their locations, managing their location of onsite or external hosting, and viewing of the files regardless of their hosting location.
+The information about the uploaded files should be stored in the `wtkFiles` data table.  To make them **private** simply set the `TempDownload` column to 'Y'.  This works regardless of whether file is stored locally or externally (AWS S3 or Cloudflare R2).
+**File Definition**
+<code SQL>
+-- This is the MySQL table structure; a PostgreSQL version is available as well of course
+CREATE TABLE `wtkFiles` (
+  `UID` int UNSIGNED NOT NULL AUTO_INCREMENT,
+  `AddDate` timestamp NOT NULL default CURRENT_TIMESTAMP,
+  `UserUID` int UNSIGNED COMMENT 'who uploaded',
+  `TableRelation` varchar(30) NOT NULL,
+  `ParentUID` int UNSIGNED COMMENT 'associated with TableRelation',
+  `Description` varchar(120) NULL,
+  `FilePath` varchar(30) NULL,
+  `OrigFileName` varchar(110) NULL,
+  `NewFileName` varchar(12) NULL,
+  `FileExtension` varchar(20) NOT NULL,
+  `FileSize` int,
+  `CurrentLocation` char(1) DEFAULT 'L' COMMENT 'L for local, A for AWS S3, C for Cloudflare R2',
+  `ExternalStorage` char(1) DEFAULT 'N' COMMENT 'Y for AWS, Cloudflare, etc.',
+  `TempDownload` ENUM('N','Y') DEFAULT 'N' COMMENT 'Y for private bucket requiring internal download to view',
+  PRIMARY KEY (`UID`),
+  KEY `ix_wtkFiles_TableRelation` (`TableRelation`,`ParentUID`),
+  KEY `ix_wtkFiles_ExternalStorage` (`ExternalStorage`),
+  FOREIGN KEY (`UserUID`) REFERENCES wtkUsers(`UID`)
+) ENGINE=InnoDB DEFAULT CHARSET=latin1 AUTO_INCREMENT=1;
+</code>
+If you want all your files to be "Private" then you should change the file definition to have `TempDownload` default to 'Y'.
+Likewise if you plan on having the majority of your files stored externally you should default `ExternalStorage` to 'Y'.  It is recommended to always keep `CurrentLocation` defaulting to 'L'.  That way your website or app will upload the file to your local server initially.  The wtk/cron.php file has code available which checks all `wtkFiles` that are both `CurrentLocation` = 'L' AND `ExternalStorage` = 'Y'.  Those files it uploads to your Cloudflare R2 or AWS S3, then it updates the `wtkFiles`.`CurrentLocation` to 'C' (or you can change it to 'A') and deletes the file from your local server.
+Many websites require having file uploads associated with different purposes or parent tables.  The `wtkFiles` has `TableRelation` and `ParentUID` to track that exact need.  This way if you need file uploads tracked for your `wtkUsers` and your `pets` data tables, they can both use `wtkFiles` and just set the `TableRelation` to 'wtkUsers' or 'pets' respectively.
+The `FilePath` stores the location the file is in.  When on your server this will be a relative path.  When on an external storage like S3 or R2 the default code instead uses the 'YY/Mon' of the upload date for the file.  That's so you do not exceed any folder maximums and it also makes it much easier to find in S3 or R2.  For example if the file was uploaded 2022-02-14 then the external path would be /22/Feb/{NewFileName}.
+===== Private File Locations =====
+When files should not be publicly accessible they should be stored in a secure location.  This can be external storage like AWS S3 or Cloudflare R2.  Or it could simply be in a folder on your server that is not accessible from the website.
+When a file is marked as `wtkFiles`.`TempDownload` = 'Y', that tells Wizard's Toolkit `viewFile.php` file to copy them from their secure location into the /exports folder.  During this process it renames them and encodes the names.
+==== Private File Cleanup ====
+The CRON job (wtk/cron.php) which should run once every minute will delete any files in the /exports folder which are more than 15 minutes old.  You can change that of course to wait longer before deleting or to delete them faster.  The /exports folder is used for this and for exporting reports to CSV or XML.
+=== AWS S3 and Cloudflare R2 ===
+Generally speaking, the external storage solution by Cloudflare R2 costs dramatically less than Amazon's S3 service.  Cloudflare R2 uses the exact same Amazon S3 SDK so if you have S3 working migrating to Cloudflare is extremely easy code-wise.  If you have external files in S3 and want to migrate them to Cloudflare R2, the Wizard's Toolkit contains pages to manage that for you.  All you need to do is modify the pathing logic based on your current S3 path rules.
-===== More Coming Soon =====
+Cloudflare R2 advantages over AWS S3
+  * less cost than AWS
+  * no download costs!
+  * no glacier storage so all files download fast, even very old ones
+  * huge company - very reliable
+  * fast uploads due to CDN
-More technical details will be added soon.
+Message Alec for more details.