Today you can do this with CSS - setting something like
button.cord-tertiary[data-cord-button=add-attachment] {
display: none
}
Would prevent the button from appearing anywhere. We're working on a version where you can directly control your own "Toolbar" implementation - the set of buttons available to a user in the Composer, but it's not ready yet.