=============================================================================================================

Tuesday, May 10, 2011

microSD ATmega32 Data-Logger



Hi friends,

aim of this project is to present a way to store a large quantity of data into microSD card in files with FAT32 format. Here, ATmega32 is used for data collection and microSD interface. The data is received from in-build 8-channel ADC of ATmega32. One channel is used for reading temperature from LM35 sensor and remaining channels are used for simply reading voltages and storing them.

This project can be used to interface 8 different sensors with ADC of ATmega32, similar to the LM35 used here. The data is stored in CSV (comma separated values) format, which can be read using a PC/Laptop with Microsoft Excel or other compatible software. A snapshot of the excel file is given later in this post.

This project is an example of how to use the microSD FAT32 library presented in my earlier post. In that post, the files were created using hyper-terminal and entering data with the PC keyboard, since that demonstrates the file creation and it's easy to debug. But many users have requested to make the file creation independent of the terminal, done inside the microcontroller, so I'm showing here how to use those functions independent of terminal. If you have directly landed on this page, it would be more helpful if you visit the original post first as it would be a better starting place for learning SD or FAT32 functions.

Here is the schematic (click on the images for larger view or download PDF):

The project contains RTC interface (for date and time storage), RS232 (for connection with PC) and a microSD module. Here, the hyper-terminal connection is required only for setting RTC date and time. Once the date/time are set, the RS232 connection is not required anymore for normal data-logging operation (It can be used for debugging purpose if there is a problem).

The microSD module used here is from eXtreme Electronics.
The module is shown in the figure here. Other than the microSD socket, this low-cost module also contains on-board 3.3v regulator for the microSD card, a 5v-3.3v level converter and other safety features required for the card. This module is used here as it provides a stable interface and makes the the card compatible with 5v supply and 5v signals of microcontroller.

The module is available at: http://store.extremeelectronics.co.in/MicroSD-TF-Module.html

The schematic also shows two LEDs and a push-button. The LEDs are used for indications of power and recording and the push-button is used to start-stop recording.

Operation of the circuit:
-----------------------------------------------
For setting RTC date/time (or for debugging mode):
  • Connect the microSD module, insert the microSD card
  • Connect the RS232 cable with the circuit. Set-up hyper terminal with 19200 baud, no parity, 8-bit data, 1 stop-bit and flow-control as 'None'
  • Connect the power cable and power on the circuit while keeping the push-button pressed
  • Green LED will glow in the circuit board
  • A menu will be displayed on the Hyper terminal as shown in the figure below. Select desired option and follow the displayed instructions
  • When date/time is set or debugging done, select option '0' to come out of the menu and start functioning a s data-logger
  • At this point, the RS232 cable can be removed


Operation as Data-Logger:
  • Connect the power cable and power on the circuit
  • Green LED will glow
  • Whenever the data-logging is required, press the push-button
  • Red LED will glow, indicating that the recording has started
  • To stop recording, press the push-button again, recording will stop and red LED will turn off
  • Files stored in the card can be read using a PC card-reader or using hyper-terminal with the circuit started in debugging mode

The operation is very simple as it uses just one push-button and an LED indication. In case of any error in accessing the card, red LED will blink continuously. In such a case, you can start circuit in debug mode (with terminal) and see the error messages.

Files are stored with the date as a name and .CSV extension. For example, data-logging done on 10 May 2011 would be stored in "10052011.CSV" file. Since the date is the name of file, everyday a single file is created and all the data recording done in a day goes into single file, no matter how many times the recording is stopped/started. First column of the file shows date, second shows time and next 8 columns show data from the 8 channels.

A file created during testing is shown in the figure below, where 5 sec interval was set for measurements (click on the image to enlarge it). Here channel-0 was used for LM35 temperature sensor, and remaining channels measure voltage. 5v was connected to channel-1 and 3v Li cell was connected to channel-3 (Channel 2 & 4 show some small voltages due to noise from voltages connected to nearby channels, which can be corrected by using bypass caps).


The interval between two measurement cycles is defined in main.c file, which can be set as per the user requirement. Basically, the program forms a dataString in every measurement cycle and appends this string to the file, if the file already exists or it creates a new file (for example, during the first recording in a day). You may go through the comments in the source code file for more info.

Note: Make sure that RTC circuit is properly connected, otherwise the code will simply hang waiting for receiving date & time from RTC

Download project files
--------------------------------------
The source code is written in AVR-GCC format using winAVR with AVRStudio-4, complete AVRStudio project folder can be downloaded from here:

- Download Source Code

Download schematic:
- Schematic (PDF)
- Schematic (EAGLE)

References:
--------------
Visit my earlier post for the references on SD card & FAT32 which are given at the end of the post

Wednesday, December 8, 2010

microSD FAT32 testing using Visual C++


Hi friends,

this post presents a way for testing and learning the FAT32 system on microSD/ SDHC cards without building the hardware with microcontroller, thanks to Henry Yiu.

This project uses the FAT32 library available in my previous post, but does away with the microcontroller part. So, you can use this code with a PC and USB card reader and get insight of the FAT32 data structure by accessing data from the card in raw as well as in FAT format.

This code can be used as a confidence building step for those who are not comfortable with directly building the hardware part. Once the FAT is understood, one can go ahead with development of hardware and customization of the software.

This may look like hardcore software thing, but it's not so. It can be tested by anybody with little knowledge of programming and determination to understand the FAT / sd card formats.

Step by step procedure for testing of the code is explained later.

Here are the Henry's words for the project:
-----------------------------------------------------------------------
When trying to build a microcontroller to access an SD card, I came across this web site which details the hardware and software of how to use an Atmel AVR microcontroller to access an SD card. But I wanted to test out the FAT32 file system code before actually building the hardware. Therefore, I wrote this code to run on Visual C++ to enable me to test the FAT32 code without actually building the hardware.

Tracing the code can reveal many compatibility problems between how the official FAT32 documentation and how Microsoft Windows actually implement the FAT32 system. There are a few interesting things I found for Windows XP. Maybe you can find more using various trace methods and many different brands of SD cards.

The things I found are, for example:

1. After adding or deleting files in Windows XP, the FSinfo next free cluster upper 16-bit is always zero.
2. Adding new files in Windows XP updates the directory structure, but does not set the next entry to empty.
3. Deleting files in Windows XP does not set the FSinfo structure next free cluster to a lower numbered cluster of the deleted file.

Recommended softwares (All the three are available for free):
---------------------------------------------------------------------------------
Microsoft Visual C++ Express

HxD Hex Editor and Disk Editor
The DevCon command-line utility functions

How to use: Only five files:
MAIN.CC, SDCARD.CC, FAT32.CC, FAT32.H, GLOBAL.H.

Use Microsoft Visual C++ Express to build these files with the default built option. Then you can step through the program to learn the boot record structure, the FAT table, and how clusters are linked together. You can also use HxD Hex Editor (Extras -> Open Physical Disk) to study the SD card content. Since data is cached in Windows, you need to unplug and plug the USB card reader for write or delete command to be visible in Windows. The DevCon utility allows you to do so without the need to actually do a hardware plug and unplug. Then you can test the FAT32 system using a batch file.

Enjoy
BR - Henry Yiu

You can download the complete project folder from here:
microSD-FAT32 with Visual C++

Many thanks to Henry for allowing to share his work

Here is the step-by-step procedure of how to use the code:
----------------------------------------------------------------------------------------
(well, this assumes that you have already done some basic background study on FAT32 and microSD cards, may be by reading a few articles on them, since some theoretical knowledge would really help in making sense of the results you get)

(the folder which you download already contains the .exe file, but we'll assume that it is not there, to make the procedure complete)

1. Install and open VC++ express

2. Create a new project (select 'empty project' when asked). Enter the location and project-name. Here we'll select project-name as 'sdcard', and location as desktop

3. Project window will open. Now, go back to desktop and copy the 5 files (MAIN.CC, SDCARD.CC, FAT32.CC, FAT32.H, GLOBAL.H) from the downloaded folder, into the folder created by the VC++

4. Go back to VC++ window and add the header files and source files into respective categories displayed at left side window panel. The panel will now look like this (here main file is open):


5. Now, from the top menu-line, under 'debug', select 'build solution'. This will build the project files. The build window at the bottom of the page is shown in this figure:


6. Now, select 'start debugging' under the same 'debug' menu. This is to make sure that the .exe file is generated. Okay, now you are ready with the .exe file. It will be stored in the 'debug' folder inside the project directory. In our case directory is 'sdcard' on the desktop.

7. It's time to test the .exe file. So, connect the USB card reader, insert a microSD card (make sure the card is already formatted with FAT32, you can also store some files for checking the read feature of the code)

8. Open windows command prompt. In the command prompt, go to the project directory, 'debug' folder. Now, type 'sdcard' or 'sdcard.exe'. This will display the available options. This is described in this figure:

9. Using the command shown in the above figure, you can select 'read', 'write', 'delete' etc. options, as per the command format. After every command executed, it would display 'success!'. If any error, it would display an error message. Some of the file operations are shown here:


10. Once you have performed some operations in last step by creating, deleting files, you need to remove the card and re-insert for checking out the files. (The re-inserting thing is not required if you use the DevCon utility as suggested by Henry). The files which are created will contain random data, as this code is mainly to understand FAT. What is important is that windows should not display that the file is corrupt. This figure shows the results of operations we did in the last step, by opening the card in windows after re-insert:


11. Now, the next step is to see how to use the HxD Hex Editor to understand what we did in the previous steps: Install and open the HxD. From the top menu-line, under 'extras', select 'open disk', select the drive letter of the card and press OK, or select 'physical disks' & then 'removable disk1'. This will open the card memory area, starting with the first sector of the card. Here in the first sector, you can see how the boot sector or MBR (master boot record) of your card looks like.

12. Using the 'find' menu, search for the text string, the name of any file stored in the card. This will take you to the FAT (File Allocation Table) area of the card. (you need to use 'search' option, as sometimes the FAT sector will be located far away and you'll be tired of scrolling down. Alternatively, using the boot-sector data, you can manually calculate the FAT's starting sector number and directly go there by entering the sector number in the box at upper right corner). Here you can see how files are arranged in FAT, by their names, starting cluster numbers etc. Following figure shows the occupied part of FAT and shows location of 'newfile.txt', which we created in step 9.


This is really useful in understanding the effects of creation, deletion of files on FAT table & FSinfo sector, to gain a real insight of the FAT32. And this understanding helps a lot while debugging your code when you make the actual hardware.

It also helps understanding the differences in file handling by the microcontroller FAT32 code and the windows, as you can check the card data in HxD after using either of them. It'll help in debugging by bridging in the troublesome differences.

Here are the additional testing suggestions by Henry:
------------------------------------------------------------------------
In order to start debugging, you will need to enter command lines. You can either change the #if1 to #if0 in main(), or use the Visual C++ command argument input: In Visual C++, select the "Project", then "SDCard Properties", then select "Debugging", and then "Command Argunments", then enter the command. For example:" -w2 e: test1.txt".

To trace the detail of the FAT32 system, you would go to the end of the getBootSectorData() function and select "Run to Cursor". Then you would record these two important sector numbers:

unusedSectors + reservedSectorCount = starting sector of the FAT table
firstDataSector = starting sector of the data area

Now you can open the HxD hex editor to see how these two sectors and how they are being changed by the program once you start stepping through the program. You can press F10 to single step, or F11 to step into a function, or right click on a line and set a breakpoint, then press F5 to run to the breakpoint. You need to press F5 in HxD hex editor to refresh the display in order to see any changes the program made to the SD card.

Once you have master this procedure, you can try doing file transfer between Windows XP and the program, for example, writing a file in Windows XP and reading it back by the program etc. You can also use HxD hex editior to view how Windows XP changes the card when it create or delete files on the card.


Enjoy the testing, It's really a fun!!!

Saturday, January 31, 2009

SD/SDHC Card Interfacing with ATmega8 /32 (FAT32 implementation)


Hi friends,
Here is my project on interfacing of SD Card (microSD). microSD cards are available very cheap nowadays, a great option for having a huge memory in any embedded system project. It is compatible with SPI bus, so the interfacing is easy. SD card adapters are also easily available in market, one can easily make a bread-board adapter by soldering few pins on it. Following figures show the SD card pin-out & the bread-board adapter design by soldering 7-pins of a breakout header on the microSD adapter (Click on images for larger view).












I had started this project with 1GB microSD card from SanDisk (later on tested with transcend cards also). The microcontroller is AVR ATmega8 or ATmega32 running at 8Mhz internal clock. MAX232 is used to interface the circuit with PC for monitoring the data. A 3.3v supply is used for powering the mega8, microSD and max232 (though the specified supply for max232 is 5v, it works comfortably at 3.3v).7 pins of the microSD are used here, shown in the figure of pin-out.

Schematic for ATmega8 is shown here (updated on 10 May 2010, SD series resistors are removed, as they were limiting the speed of SPI bus. 51k pullups are added on CMD/DAT lines. This gives better stability with different cards. Also, two 3.6v zeners are added to protect SD in case when the ISP programmer voltage levels are of 5v. these diodes are not required if your programmer has settings for 3.3v output)
(Note: VCC & GND pins of MAX232 are not shown in the schematic, but they must be connected in the actual hardware)
Following is the schematic for ATmega32, without RTC (updated on 10 May 2010):


Following is the schematic for ATmega32, with RTC (added on
17 May 2010; CS pin correction, PB4 instead of PB1, done in Mar 2014). Here two supply voltages are used, 3.3v for SD & 5v for remaining ICs.


The aim of this project was to learn interfacing of SD card and to understand the data transfer in raw format as well as in FAT32 format. I started with raw data transfer, sending some data to any block of the microSD, reading a block of it, reading and writing multiple blocks, erasing multiple blocks. All this in raw format. I used RS232 for viewing the data read by microcontroller from SD card. The uc sends the data to HyperTerminal. Similarly, to write data to card, the data was fed thru HyperTerminal, by typing some text.

Once raw data transfer achieved, I formatted the card with windowsXP (FAT32) and loaded it with some text files, directories and other files (all stored in root directory of the card). After that I wrote the FAT32 routines to read files, get file list (using HyperTerminal again), finding the total/free memory of card. All this data is sent to HyperTerminal by the uc.

Following is the HyperTerminal window showing different options:
Options 0 to 4 are low level functions dealing with raw data. If you use option 0, 1 or 3, you may have to reformat the card before using the FAT32 routines.
0: Erases selected number of blocks strating from selected block
1: Writes data to specified SD block address. Data to be entered in HyperTerminal using PC keyboard
2: Readss data of specified SD block address. Data is displayed on HyperTerminal window
3. Writes selected number of blocks strating from selected block
4. Reads selected number of blocks strating from selected block

Here, the multiple-block functions related to options 3 & 4 are disabled due to memory constraint as that time mega8 was used for testing and these functions are not required for FAT32 testing. While testing with mega32, options 3 & 4 can be enabled by removing a macro (#define FAT_TESTING_ONLY) defined in SD_routines.h.

Options 5 to 9 are related to FAT32 . Only short file names are supported right now, 8byte name+3bytes extension. If you store a long name file in SD, it will be displayed by these routines in short name format only.
For testing these options, format the card with FAT32 file system and store some directories and text files (because text files can be read & checked thru HyperTerminal).

5: Displays list of available directories and files with size (in the root directory of the card)
6: Reads a specified file and displays the file contents on HyperTerminal
7: Create/Append file with specified name, enter text from HyperTerminal
8: Deletes any existing file with specified name
9: Displays total & free memory of the card (using FSinfo sector of the SD card)

Following figures show the HyperTerminal window when options 5 & 9 are selected:
(These figures show menu from Ver2.3 or earlier. Menu style is changed from Ver_2.4 onwards, which is shown in the update history)

Note: HyperTerminal is used here at 19200 baudrate, No parity, Flow Control 'none'.

This project needs very few components and can be done easily at home. Try it out!


Download the source code files from here
:
Download here the zipped source code files modified for mega32, written in winAVR-AVRStudio.

Version 2.4.1 (RTC added for Date/Time entries) 17 May 2010/24 Apr 2011
Version 2.3 (SDHC support added) 09 May 2010
Version 2.2 (No SDHC support) 13 Sep 2009

Download EAGLE schematic file of Ver 2.4

Download/view source code files V2.1 (for ATmega8):
Following files are compiled using winAVR inside AVRStudio. This Version does not support SDHC cards. Also, append file feature is not available.
1. SD_main.c
2. SD_routines.c & SD_routines.h
3. FAT32.c & FAT32.h (Ver 2.1, last updated-13 Sep 09)
4. SPI_routines.c & SPI_routines.h
5. UART_routines.c & UART_routines.h
6. Makefile
7. HEX file (Ver 2.1, last updated-13 Sep 09)

Please put up a comment or mail me if you find a bug in the code. The updates are because of valuable suggestions & comments from the users of this code. Thanks a lot!!


Buy from Farnell (Element14): ATmega32 (India), ATmega32 (USA)


Related posts:
You can visit my post of microSD ATmega32 Data-Logger, which uses modified FAT32 library for automatically creating files without using hyper terminal.
If you want to test FAT32 functions or learn more without making the microcontroller hardware, you can visit my post here: microSD-FAT32 using Visual C++)

Update History-----------------------------------------------------------------------------
Version 2.4:

- Real Time Clock circuit support is added for time & date entries in the files. Now the current date of file creation and file update will be entered in the FAT table (can be viewed by checking file 'properties' using a PC)
(The RTC will also be useful in data-logging with time-stamp)
- Three more options added in the Hyper Terminal menu for displaying or updating RTC date & time. New menu is shown in the above figure.

Ver 2.4.1:
- Same as 2.4, with a bug fix for RTC: 'twi_init' function added to define I2C clock freq. 100K@ 16MHz (50k@8MHz). This was taking default values earlier, which was as high as 500K@8MHz, not desirable

(Note: Version 2.2, 2.3 & 2.4 are tested on ATmega32, but they can be adopted to any controller having RAM >= 1KB and Flash >= 16KB)
Current memory usage (Ver_2.4): Flash: 12908 Bytes; RAM: 700 Bytes (appx.);

Version 2.3:
- Support for SDHC cards added (tested with SanDisk & Transcend microSD and microSDHC cards). The initialization sequence and command formats are modified.
- A bug which was causing the program flow to go into infinite loop if the character number 512 in a sector was a CR (Carriage Return, '\r'), in the writeFile function. Thanks to David & Piotr M. who pointed it out in the comments.
- Code is also tested successfully at 16MHz clock (8MHz SPI clock) with for SD/SDHC cards.

Follwing are the Hyper Terminal windows showing card detection (One window shows baudrate as 38400, that was while testing for higher clock speeds, current code still uses 19200 baud and 8MHz internal clock of Mega32).








Version 2.2:
- Append file feature added. 'createFile' function replaced with writeFile, which looks for the filename first, if the given file name doesn't exist then it creates new file and writes data, but if the file already exists, then it opens it and appends the entered data.
- A bug removed which was giving error related to use of 'LONG'
- The FAT32.c & .h files are updated on 13 Sep 09 to remove a bug which was limiting file size to 32MB (Thanks to Kun-Szabo Marton who pointed it out)

Data transfer rate: 1 raw data block (512 bytes) takes 4.15ms for reading or writing (123.37 KBytes/s) at current 4 MHz SPI clock rate. If you have flash more than 8k, you can declare the SPI_receive() and SPI_transmit() functions as 'inline' functions. This will increase the transfer rate to 140 KBytes/s. These transfer rates can be further increased by using a 16MHz crystal (8 MHz SPI clock). FAT32 file reading is done at 78 to 91 KBytes/sec.

Version 2.1:
- A bug removed which stopped creating new files after 32*8 files in the root directory
- The root directory was unnecessarily getting expanded by one cluster whenever a file was created. Fixed in the new version
- Also, the fixed cluster size of 8 sectors is removed, this version will support other cluster sizes as well

Version 2.0:
- Support added for SD cards having first sector as MBR rather than the boot sector
- createFile and deleteFile functions added
- A bug fixed in reading files stord at far locations in memory also correction made to accept 8+3 char file name (by mistake, it was taking 7+3 earlier)
- FSinfo sector used for storing total free cluster count &
next free cluster number for faster file access
- Instant freeMemory display (earlier it was taking more than 30secs) using FSinfo sector. FSinfo sector is updated now whenever a file is created or deleted
- File memory size display in decimal, like windows (earlier it was in hex)
- Raw SD functions multiple block read and write, which are not required for FAT32, are disabled using FAT32_TESTING_ONLY macro for getting extra space required by createFile & deleteFile changes (you can activate it if you have more than 8k flash) Right now flash is 99.9% full
- Clock speed raised from 1Mhz to 8 MHz, new SPI speed (after SD initialization): 4MHz instead of 500K & baudrate: 19200 (for HyperTerminal) instead of 4800
-------------------------------------------------------------------------------

References:
1. Microsoft's FAT32 specification document
2. SanDisk SD mnual v1.9
3. SD-Simplified Physical Layer Spec. Ver 4.10
4. F. Foust's Application Note on SD
5. FAT32 Structure info, includes MBR details
6. Simple FAT32 Structure explanation
 
7. Chan's FAT library
8. SD Association's Website for further info

Regards,
ccd@dharmanitech.com