Originally I came to the forum to ask for help integrating MATLAB and SB3, but I ended up solving it. Here are some syntax tips that I've found for controlling SB3 with MATLAB 2017b.

To call a part file from MATLAB:
system('"C:\Program Files (x86)\ShopBot\ShopBot 3\Sb3.exe" "C:\Users\username\Documents\zero.sbp "," ",4," ",0," "2," "," "&');

To send signals back to MATLAB while the part file is running:
Insert a line into your part file to set appropriate output flags. You can watch the output flags during the part file to see which ones your file already uses (I know 1 and 2 might be used for the spindle?). For example, to activate flag 3, issue the command
SO, 3, 1
and to deactivate it, issue
SO, 3, 0
In MATLAB, you can read those flags using Windows Registry values.
output = str2double(winqueryreg('HKEY_CURRENT_USER', 'Software\VB and VBA Program Settings\Shopbot\UserData', 'OutPutSwitches'));
This will return a number that represents the output flags as a binary encoded sequence. i.e. flag 1 is 1, flag 2 is 2, flag 3 is 4, flag 4 is 8, and it'll return the sum of those flags, so if flag 1 and 4 are active then "output" will be 9.
You can tell MATLAB to continually watch the output flags by creating a timer in MATLAB. For example, if you're putting this in a GUI, you can use the following code in your OpeningFcn and then put your winqueryreg function inside of the function "check_flags."
handles.timer1 = timer('ExecutionMode', 'fixedRate', 'Period', 0.1, 'TimerFcn', {@check_flags, hObject},'ErrorFcn', {@resetTimer});
start(handles.timer1);


I hope somebody finds this helpful! The programming guide suggests a fourth input in the registry query that you don't need in MATLAB but maybe need in Basic, so getting the flags out can be kind of confusing. Also if winqueryreg doesn't work, your registry values might be in a different place. Open up regedit.exe and search for 'OutPutSwitches' which probably won't be anywhere else.